2024-03-18T00:58:50.799 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-18T00:58:50.799 DEBUG:teuthology.run:Teuthology command: teuthology --archive /home/teuthworker/archive/pdonnell-2024-03-17_23:17:27-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7608561 --verbose --name pdonnell-2024-03-17_23:17:27-fs-wip-batrick-testing-20240314.183243-distro-default-smithi --description fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} --owner scheduled_pdonnell@teuthology -- /home/teuthworker/archive/pdonnell-2024-03-17_23:17:27-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7608561/orig.config.yaml 2024-03-18T00:58:50.855 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-18T00:58:50.951 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/pdonnell-2024-03-17_23:17:27-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7608561 branch: wip-batrick-testing-20240314.183243 description: fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} email: pdonnell@redhat.com first_in_suite: false job_id: '7608561' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi meta: - desc: 'setup ceph/quincy ' name: pdonnell-2024-03-17_23:17:27-fs-wip-batrick-testing-20240314.183243-distro-default-smithi no_nested_subset: false nuke-on-error: true os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-batrick-testing-20240314.183243 ceph: conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: mon pg warn min per osd: 0 mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - POOL_APP_NOT_ENABLED - overall HEALTH_ - Replacing daemon - deprecated feature inline_data - mons down, quorum - OSD_DOWN - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - MON_DOWN sha1: 3d78f5ba07bf18880172c07d4d7b4ac00c9d30af ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: 3d78f5ba07bf18880172c07d4d7b4ac00c9d30af kclient: syntax: v1 selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 - scontext=system_u:system_r:getty_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-batrick-testing-20240314.183243 sha1: c382027d5c66ef4e0aa1131ea3e8080c658d3184 owner: scheduled_pdonnell@teuthology priority: 75 repo: https://git.ceph.com/ceph-ci.git roles: - - host.a - client.0 - osd.0 - osd.1 - osd.2 - - host.b - client.1 - osd.3 - osd.4 - osd.5 seed: 4770 sha1: 3d78f5ba07bf18880172c07d4d7b4ac00c9d30af sleep_before_teardown: 0 subset: 498/512 suite: fs suite_branch: wip-batrick-testing-20240314.183243 suite_path: /home/teuthworker/src/github.com_batrick_ceph_c382027d5c66ef4e0aa1131ea3e8080c658d3184/qa suite_relpath: qa suite_repo: https://github.com/batrick/ceph suite_sha1: c382027d5c66ef4e0aa1131ea3e8080c658d3184 targets: smithi029.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGW25UlkH/bXu4KCYGfVm5U2VBzpkQLaPDsijUdeDame6Fwc6a3NjBriD+CKqSrhlp3TjuC2OL0SiPDlE1QnNwk= smithi090.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHzv2Sz8HwWmBAuxDXa5G3WXiAPTu2EWfpwaaLQLCCw2tsnu16FDoyG4iehdAol9ADDbmrI0DVahcmM7kKti0Z8= tasks: - install: branch: quincy exclude_packages: - ceph-volume - print: '**** done install task...' - cephadm: cephadm_branch: quincy cephadm_git_url: https://github.com/ceph/ceph conf: osd: osd_class_default_list: '*' osd_class_load_list: '*' image: quay.ceph.io/ceph-ci/ceph:quincy roleless: true - print: '**** done end installing quincy cephadm ...' - cephadm.shell: host.a: - ceph config set mgr mgr/cephadm/use_repo_digest true --force - print: '**** done cephadm.shell ceph config set mgr...' - cephadm.shell: host.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: host.a: - ceph fs volume create cephfs --placement=4 - ceph fs dump - cephadm.shell: host.a: - ceph fs set cephfs max_mds 2 - cephadm.shell: host.a: - ceph fs set cephfs allow_standby_replay true - cephadm.shell: host.a: - ceph fs set cephfs inline_data true --yes-i-really-really-mean-it - cephadm.shell: host.a: - ceph fs dump - ceph --format=json fs dump | jq -e ".filesystems | length == 1" - while ! ceph --format=json mds versions | jq -e ". | add == 4"; do sleep 1; done - fs.pre_upgrade_save: null - kclient: null - print: '**** done client' - parallel: - upgrade-tasks - workload-tasks - cephadm.shell: host.a: - ceph fs dump - fs.post_upgrade_checks: null teuthology: fragments_dropped: - /cephfs/home/pdonnell/src/github.com_batrick_ceph_wip-batrick-testing-20240314.183243/qa/suites/fs/upgrade/mds_upgrade_sequence/tasks/3-upgrade-mgr-staggered.yaml meta: {} postmerge: [] variables: fail_fs: false teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-17_23:17:27 tube: smithi upgrade-tasks: sequential: - cephadm.shell: env: - sha1 host.a: - ceph config set mgr mgr/orchestrator/fail_fs false || true - cephadm.shell: env: - sha1 host.a: - ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force - ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force - ceph config set global log_to_journald false --force - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 - cephadm.shell: env: - sha1 host.a: - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph fs dump; ceph orch upgrade status ; ceph health detail ; sleep 30 ; done - ceph orch ps - ceph orch upgrade status - ceph health detail - ceph versions - echo "wait for servicemap items w/ changing names to refresh" - sleep 60 - ceph orch ps - ceph versions - ceph versions | jq -e '.overall | length == 1' - ceph versions | jq -e '.overall | keys' | grep $sha1 user: pdonnell verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 workload-tasks: sequential: - workunit: clients: all: - suites/fsstress.sh 2024-03-18T00:58:50.951 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_batrick_ceph_c382027d5c66ef4e0aa1131ea3e8080c658d3184/qa; will attempt to use it 2024-03-18T00:58:50.952 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_batrick_ceph_c382027d5c66ef4e0aa1131ea3e8080c658d3184/qa/tasks 2024-03-18T00:58:50.952 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-18T00:58:50.953 INFO:teuthology.task.internal:Checking packages... 2024-03-18T00:58:50.975 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '3d78f5ba07bf18880172c07d4d7b4ac00c9d30af' 2024-03-18T00:58:50.976 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-18T00:58:50.976 INFO:teuthology.packaging:ref: None 2024-03-18T00:58:50.976 INFO:teuthology.packaging:tag: None 2024-03-18T00:58:50.976 INFO:teuthology.packaging:branch: wip-batrick-testing-20240314.183243 2024-03-18T00:58:50.976 INFO:teuthology.packaging:sha1: 3d78f5ba07bf18880172c07d4d7b4ac00c9d30af 2024-03-18T00:58:50.976 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-batrick-testing-20240314.183243 2024-03-18T00:58:51.248 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2162.g3d78f5ba 2024-03-18T00:58:51.249 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-18T00:58:51.258 INFO:teuthology.task.internal:no buildpackages task found 2024-03-18T00:58:51.258 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-18T00:58:51.265 INFO:teuthology.task.internal:Saving configuration 2024-03-18T00:58:51.283 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-18T00:58:51.289 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-18T00:58:51.313 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi029.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-03-17_23:17:27-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7608561', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-18 00:53:09.031076', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGW25UlkH/bXu4KCYGfVm5U2VBzpkQLaPDsijUdeDame6Fwc6a3NjBriD+CKqSrhlp3TjuC2OL0SiPDlE1QnNwk='} 2024-03-18T00:58:51.335 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi090.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-03-17_23:17:27-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7608561', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-18 00:53:09.029423', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHzv2Sz8HwWmBAuxDXa5G3WXiAPTu2EWfpwaaLQLCCw2tsnu16FDoyG4iehdAol9ADDbmrI0DVahcmM7kKti0Z8='} 2024-03-18T00:58:51.335 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-18T00:58:51.341 INFO:teuthology.task.internal:roles: ubuntu@smithi029.front.sepia.ceph.com - ['host.a', 'client.0', 'osd.0', 'osd.1', 'osd.2'] 2024-03-18T00:58:51.341 INFO:teuthology.task.internal:roles: ubuntu@smithi090.front.sepia.ceph.com - ['host.b', 'client.1', 'osd.3', 'osd.4', 'osd.5'] 2024-03-18T00:58:51.341 INFO:teuthology.run_tasks:Running task console_log... 2024-03-18T00:58:51.414 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fcfb3c02160>, signals=[15]) 2024-03-18T00:58:51.415 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-18T00:58:51.421 INFO:teuthology.task.internal:Opening connections... 2024-03-18T00:58:51.421 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi029.front.sepia.ceph.com 2024-03-18T00:58:51.423 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi029.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-18T00:58:51.528 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi090.front.sepia.ceph.com 2024-03-18T00:58:51.529 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-18T00:58:51.618 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-18T00:58:51.625 DEBUG:teuthology.orchestra.run.smithi029:> uname -m 2024-03-18T00:58:51.643 INFO:teuthology.orchestra.run.smithi029.stdout:x86_64 2024-03-18T00:58:51.643 DEBUG:teuthology.orchestra.run.smithi029:> cat /etc/os-release 2024-03-18T00:58:51.699 INFO:teuthology.orchestra.run.smithi029.stdout:NAME="CentOS Stream" 2024-03-18T00:58:51.699 INFO:teuthology.orchestra.run.smithi029.stdout:VERSION="9" 2024-03-18T00:58:51.700 INFO:teuthology.orchestra.run.smithi029.stdout:ID="centos" 2024-03-18T00:58:51.700 INFO:teuthology.orchestra.run.smithi029.stdout:ID_LIKE="rhel fedora" 2024-03-18T00:58:51.700 INFO:teuthology.orchestra.run.smithi029.stdout:VERSION_ID="9" 2024-03-18T00:58:51.700 INFO:teuthology.orchestra.run.smithi029.stdout:PLATFORM_ID="platform:el9" 2024-03-18T00:58:51.700 INFO:teuthology.orchestra.run.smithi029.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-18T00:58:51.700 INFO:teuthology.orchestra.run.smithi029.stdout:ANSI_COLOR="0;31" 2024-03-18T00:58:51.700 INFO:teuthology.orchestra.run.smithi029.stdout:LOGO="fedora-logo-icon" 2024-03-18T00:58:51.700 INFO:teuthology.orchestra.run.smithi029.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-18T00:58:51.700 INFO:teuthology.orchestra.run.smithi029.stdout:HOME_URL="https://centos.org/" 2024-03-18T00:58:51.700 INFO:teuthology.orchestra.run.smithi029.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-18T00:58:51.700 INFO:teuthology.orchestra.run.smithi029.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-18T00:58:51.700 INFO:teuthology.orchestra.run.smithi029.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-18T00:58:51.701 INFO:teuthology.lock.ops:Updating smithi029.front.sepia.ceph.com on lock server 2024-03-18T00:58:51.727 DEBUG:teuthology.orchestra.run.smithi090:> uname -m 2024-03-18T00:58:51.745 INFO:teuthology.orchestra.run.smithi090.stdout:x86_64 2024-03-18T00:58:51.745 DEBUG:teuthology.orchestra.run.smithi090:> cat /etc/os-release 2024-03-18T00:58:51.802 INFO:teuthology.orchestra.run.smithi090.stdout:NAME="CentOS Stream" 2024-03-18T00:58:51.802 INFO:teuthology.orchestra.run.smithi090.stdout:VERSION="9" 2024-03-18T00:58:51.802 INFO:teuthology.orchestra.run.smithi090.stdout:ID="centos" 2024-03-18T00:58:51.802 INFO:teuthology.orchestra.run.smithi090.stdout:ID_LIKE="rhel fedora" 2024-03-18T00:58:51.802 INFO:teuthology.orchestra.run.smithi090.stdout:VERSION_ID="9" 2024-03-18T00:58:51.802 INFO:teuthology.orchestra.run.smithi090.stdout:PLATFORM_ID="platform:el9" 2024-03-18T00:58:51.802 INFO:teuthology.orchestra.run.smithi090.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-18T00:58:51.802 INFO:teuthology.orchestra.run.smithi090.stdout:ANSI_COLOR="0;31" 2024-03-18T00:58:51.802 INFO:teuthology.orchestra.run.smithi090.stdout:LOGO="fedora-logo-icon" 2024-03-18T00:58:51.802 INFO:teuthology.orchestra.run.smithi090.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-18T00:58:51.802 INFO:teuthology.orchestra.run.smithi090.stdout:HOME_URL="https://centos.org/" 2024-03-18T00:58:51.802 INFO:teuthology.orchestra.run.smithi090.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-18T00:58:51.803 INFO:teuthology.orchestra.run.smithi090.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-18T00:58:51.803 INFO:teuthology.orchestra.run.smithi090.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-18T00:58:51.803 INFO:teuthology.lock.ops:Updating smithi090.front.sepia.ceph.com on lock server 2024-03-18T00:58:51.827 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-18T00:58:51.836 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-18T00:58:51.842 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-18T00:58:51.842 DEBUG:teuthology.orchestra.run.smithi029:> test '!' -e /home/ubuntu/cephtest 2024-03-18T00:58:51.845 DEBUG:teuthology.orchestra.run.smithi090:> test '!' -e /home/ubuntu/cephtest 2024-03-18T00:58:51.861 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-18T00:58:51.867 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-18T00:58:51.868 DEBUG:teuthology.orchestra.run.smithi029:> test -z $(ls -A /var/lib/ceph) 2024-03-18T00:58:51.901 DEBUG:teuthology.orchestra.run.smithi090:> test -z $(ls -A /var/lib/ceph) 2024-03-18T00:58:51.927 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-18T00:58:51.982 INFO:teuthology.run_tasks:Running task kernel... 2024-03-18T00:58:51.999 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-03-18T00:58:51.999 INFO:teuthology.task.kernel:config {'host.a': {'kdb': True, 'sha1': 'distro'}, 'host.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-03-18T00:58:51.999 DEBUG:teuthology.orchestra.run.smithi029:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-18T00:58:52.000 DEBUG:teuthology.orchestra.run.smithi090:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-18T00:58:52.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-18T00:58:52.017 DEBUG:teuthology.orchestra.run.smithi090:> uname -r 2024-03-18T00:58:52.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-18T00:58:52.018 DEBUG:teuthology.orchestra.run.smithi029:> uname -r 2024-03-18T00:58:52.074 INFO:teuthology.orchestra.run.smithi090.stdout:5.14.0-361.el9.x86_64 2024-03-18T00:58:52.074 INFO:teuthology.task.kernel:Running kernel on smithi090: 5.14.0-361.el9.x86_64 2024-03-18T00:58:52.074 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum install -y kernel 2024-03-18T00:58:52.075 INFO:teuthology.orchestra.run.smithi029.stdout:5.14.0-361.el9.x86_64 2024-03-18T00:58:52.075 INFO:teuthology.task.kernel:Running kernel on smithi029: 5.14.0-361.el9.x86_64 2024-03-18T00:58:52.075 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum install -y kernel 2024-03-18T00:58:54.573 INFO:teuthology.orchestra.run.smithi090.stdout:CentOS-9 - AppStream 13 MB/s | 6.7 MB 00:00 2024-03-18T00:58:55.216 INFO:teuthology.orchestra.run.smithi029.stdout:CentOS-9 - AppStream 13 MB/s | 6.7 MB 00:00 2024-03-18T00:58:55.973 INFO:teuthology.orchestra.run.smithi090.stdout:CentOS-9 - BaseOS 7.3 MB/s | 2.0 MB 00:00 2024-03-18T00:58:56.734 INFO:teuthology.orchestra.run.smithi029.stdout:CentOS-9 - BaseOS 5.6 MB/s | 2.0 MB 00:00 2024-03-18T00:58:58.648 INFO:teuthology.orchestra.run.smithi090.stdout:CentOS Stream 9 - BaseOS 3.5 MB/s | 8.0 MB 00:02 2024-03-18T00:59:02.966 INFO:teuthology.orchestra.run.smithi090.stdout:CentOS Stream 9 - AppStream 5.8 MB/s | 19 MB 00:03 2024-03-18T00:59:07.399 INFO:teuthology.orchestra.run.smithi029.stdout:CentOS Stream 9 - BaseOS 798 kB/s | 8.0 MB 00:10 2024-03-18T00:59:09.451 INFO:teuthology.orchestra.run.smithi029.stdout:CentOS Stream 9 - AppStream 20 MB/s | 19 MB 00:00 2024-03-18T00:59:09.767 INFO:teuthology.orchestra.run.smithi090.stdout:CentOS Stream 9 - CRB 1.9 MB/s | 6.1 MB 00:03 2024-03-18T00:59:11.345 INFO:teuthology.orchestra.run.smithi090.stdout:CentOS Stream 9 - Extras packages 35 kB/s | 16 kB 00:00 2024-03-18T00:59:13.304 INFO:teuthology.orchestra.run.smithi090.stdout:Extra Packages for Enterprise Linux 11 MB/s | 21 MB 00:01 2024-03-18T00:59:15.546 INFO:teuthology.orchestra.run.smithi029.stdout:CentOS Stream 9 - CRB 2.5 MB/s | 6.1 MB 00:02 2024-03-18T00:59:17.132 INFO:teuthology.orchestra.run.smithi029.stdout:CentOS Stream 9 - Extras packages 38 kB/s | 16 kB 00:00 2024-03-18T00:59:18.230 INFO:teuthology.orchestra.run.smithi090.stdout:lab-extras 20 kB/s | 1.7 kB 00:00 2024-03-18T00:59:19.068 INFO:teuthology.orchestra.run.smithi029.stdout:Extra Packages for Enterprise Linux 11 MB/s | 21 MB 00:01 2024-03-18T00:59:20.420 INFO:teuthology.orchestra.run.smithi090.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-18T00:59:20.421 INFO:teuthology.orchestra.run.smithi090.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-18T00:59:20.534 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-03-18T00:59:20.539 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-03-18T00:59:20.539 INFO:teuthology.orchestra.run.smithi090.stdout: Package Architecture Version Repository Size 2024-03-18T00:59:20.539 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-03-18T00:59:20.539 INFO:teuthology.orchestra.run.smithi090.stdout:Installing: 2024-03-18T00:59:20.539 INFO:teuthology.orchestra.run.smithi090.stdout: kernel x86_64 5.14.0-427.el9 baseos 6.0 M 2024-03-18T00:59:20.539 INFO:teuthology.orchestra.run.smithi090.stdout:Installing dependencies: 2024-03-18T00:59:20.539 INFO:teuthology.orchestra.run.smithi090.stdout: kernel-core x86_64 5.14.0-427.el9 baseos 21 M 2024-03-18T00:59:20.539 INFO:teuthology.orchestra.run.smithi090.stdout: kernel-modules x86_64 5.14.0-427.el9 baseos 39 M 2024-03-18T00:59:20.539 INFO:teuthology.orchestra.run.smithi090.stdout: kernel-modules-core x86_64 5.14.0-427.el9 baseos 33 M 2024-03-18T00:59:20.539 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-18T00:59:20.539 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction Summary 2024-03-18T00:59:20.540 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-03-18T00:59:20.540 INFO:teuthology.orchestra.run.smithi090.stdout:Install 4 Packages 2024-03-18T00:59:20.540 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-18T00:59:20.540 INFO:teuthology.orchestra.run.smithi090.stdout:Total download size: 100 M 2024-03-18T00:59:20.540 INFO:teuthology.orchestra.run.smithi090.stdout:Installed size: 123 M 2024-03-18T00:59:20.540 INFO:teuthology.orchestra.run.smithi090.stdout:Downloading Packages: 2024-03-18T00:59:21.232 INFO:teuthology.orchestra.run.smithi090.stdout:(1/4): kernel-5.14.0-427.el9.x86_64.rpm 15 MB/s | 6.0 MB 00:00 2024-03-18T00:59:21.857 INFO:teuthology.orchestra.run.smithi090.stdout:(2/4): kernel-core-5.14.0-427.el9.x86_64.rpm 20 MB/s | 21 MB 00:01 2024-03-18T00:59:22.466 INFO:teuthology.orchestra.run.smithi090.stdout:(3/4): kernel-modules-core-5.14.0-427.el9.x86_6 27 MB/s | 33 MB 00:01 2024-03-18T00:59:22.883 INFO:teuthology.orchestra.run.smithi090.stdout:(4/4): kernel-modules-5.14.0-427.el9.x86_64.rpm 19 MB/s | 39 MB 00:02 2024-03-18T00:59:22.883 INFO:teuthology.orchestra.run.smithi090.stdout:-------------------------------------------------------------------------------- 2024-03-18T00:59:22.883 INFO:teuthology.orchestra.run.smithi090.stdout:Total 42 MB/s | 100 MB 00:02 2024-03-18T00:59:23.765 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction check 2024-03-18T00:59:24.162 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction check succeeded. 2024-03-18T00:59:24.162 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction test 2024-03-18T00:59:24.368 INFO:teuthology.orchestra.run.smithi029.stdout:lab-extras 31 kB/s | 1.7 kB 00:00 2024-03-18T00:59:25.558 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction test succeeded. 2024-03-18T00:59:25.559 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction 2024-03-18T00:59:26.678 INFO:teuthology.orchestra.run.smithi029.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-18T00:59:26.679 INFO:teuthology.orchestra.run.smithi029.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-18T00:59:26.786 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2024-03-18T00:59:26.802 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-03-18T00:59:26.802 INFO:teuthology.orchestra.run.smithi029.stdout: Package Architecture Version Repository Size 2024-03-18T00:59:26.802 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-03-18T00:59:26.803 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2024-03-18T00:59:26.803 INFO:teuthology.orchestra.run.smithi029.stdout: kernel x86_64 5.14.0-427.el9 baseos 6.0 M 2024-03-18T00:59:26.803 INFO:teuthology.orchestra.run.smithi029.stdout:Installing dependencies: 2024-03-18T00:59:26.803 INFO:teuthology.orchestra.run.smithi029.stdout: kernel-core x86_64 5.14.0-427.el9 baseos 21 M 2024-03-18T00:59:26.803 INFO:teuthology.orchestra.run.smithi029.stdout: kernel-modules x86_64 5.14.0-427.el9 baseos 39 M 2024-03-18T00:59:26.803 INFO:teuthology.orchestra.run.smithi029.stdout: kernel-modules-core x86_64 5.14.0-427.el9 baseos 33 M 2024-03-18T00:59:26.803 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-18T00:59:26.803 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2024-03-18T00:59:26.803 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-03-18T00:59:26.803 INFO:teuthology.orchestra.run.smithi029.stdout:Install 4 Packages 2024-03-18T00:59:26.803 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-18T00:59:26.803 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 100 M 2024-03-18T00:59:26.803 INFO:teuthology.orchestra.run.smithi029.stdout:Installed size: 123 M 2024-03-18T00:59:26.803 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2024-03-18T00:59:27.982 INFO:teuthology.orchestra.run.smithi029.stdout:(1/4): kernel-5.14.0-427.el9.x86_64.rpm 5.3 MB/s | 6.0 MB 00:01 2024-03-18T00:59:28.364 INFO:teuthology.orchestra.run.smithi090.stdout: Preparing : 1/1 2024-03-18T00:59:28.915 INFO:teuthology.orchestra.run.smithi029.stdout:(2/4): kernel-core-5.14.0-427.el9.x86_64.rpm 10 MB/s | 21 MB 00:02 2024-03-18T00:59:29.060 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : kernel-modules-core-5.14.0-427.el9.x86_64 1/4 2024-03-18T00:59:29.199 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-18T00:59:30.082 INFO:teuthology.orchestra.run.smithi029.stdout:(3/4): kernel-modules-5.14.0-427.el9.x86_64.rpm 12 MB/s | 39 MB 00:03 2024-03-18T00:59:30.916 INFO:teuthology.orchestra.run.smithi029.stdout:(4/4): kernel-modules-core-5.14.0-427.el9.x86_6 11 MB/s | 33 MB 00:02 2024-03-18T00:59:30.916 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2024-03-18T00:59:30.916 INFO:teuthology.orchestra.run.smithi029.stdout:Total 24 MB/s | 100 MB 00:04 2024-03-18T00:59:31.838 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2024-03-18T00:59:31.985 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-18T00:59:32.175 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-18T00:59:32.285 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2024-03-18T00:59:32.285 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2024-03-18T00:59:33.676 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2024-03-18T00:59:33.677 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2024-03-18T00:59:36.535 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2024-03-18T00:59:36.922 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-18T00:59:37.046 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : kernel-5.14.0-427.el9.x86_64 4/4 2024-03-18T00:59:37.320 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : kernel-modules-core-5.14.0-427.el9.x86_64 1/4 2024-03-18T00:59:37.522 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-18T00:59:40.469 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-18T00:59:40.765 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-18T00:59:41.820 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: kernel-modules-core-5.14.0-427.el9.x86_64 4/4 2024-03-18T00:59:45.867 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-18T00:59:46.025 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : kernel-5.14.0-427.el9.x86_64 4/4 2024-03-18T00:59:51.158 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: kernel-modules-core-5.14.0-427.el9.x86_64 4/4 2024-03-18T01:00:11.958 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: kernel-core-5.14.0-427.el9.x86_64 4/4 2024-03-18T01:00:11.959 INFO:teuthology.orchestra.run.smithi090.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-18T01:00:11.959 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-18T01:00:12.045 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: kernel-modules-5.14.0-427.el9.x86_64 4/4 2024-03-18T01:00:12.520 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: kernel-5.14.0-427.el9.x86_64 4/4 2024-03-18T01:00:12.520 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : kernel-5.14.0-427.el9.x86_64 1/4 2024-03-18T01:00:12.521 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-18T01:00:12.521 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-18T01:00:12.974 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : kernel-modules-core-5.14.0-427.el9.x86_64 4/4 2024-03-18T01:00:12.975 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-18T01:00:12.975 INFO:teuthology.orchestra.run.smithi090.stdout:Installed: 2024-03-18T01:00:12.975 INFO:teuthology.orchestra.run.smithi090.stdout: kernel-5.14.0-427.el9.x86_64 2024-03-18T01:00:12.975 INFO:teuthology.orchestra.run.smithi090.stdout: kernel-core-5.14.0-427.el9.x86_64 2024-03-18T01:00:12.975 INFO:teuthology.orchestra.run.smithi090.stdout: kernel-modules-5.14.0-427.el9.x86_64 2024-03-18T01:00:12.975 INFO:teuthology.orchestra.run.smithi090.stdout: kernel-modules-core-5.14.0-427.el9.x86_64 2024-03-18T01:00:12.975 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-18T01:00:12.975 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-03-18T01:00:13.551 DEBUG:teuthology.orchestra.run.smithi090:> rpm -q kernel | sort -rV | head -n 1 2024-03-18T01:00:13.700 INFO:teuthology.orchestra.run.smithi090.stdout:kernel-5.14.0-427.el9.x86_64 2024-03-18T01:00:13.700 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-427.el9.x86_64 2024-03-18T01:00:13.700 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-427.el9.x86_64 2024-03-18T01:00:13.701 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-18T01:00:13.701 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-18T01:00:13.701 INFO:teuthology.task.kernel:Installing distro kernel on host.b... 2024-03-18T01:00:13.701 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi090.front.sepia.ceph.com, path=None, version=distro) 2024-03-18T01:00:13.701 DEBUG:teuthology.orchestra.run.smithi090:> rpm -q kernel | sort -rV | head -n 1 2024-03-18T01:00:13.820 INFO:teuthology.orchestra.run.smithi090.stdout:kernel-5.14.0-427.el9.x86_64 2024-03-18T01:00:13.820 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-427.el9.x86_64 2024-03-18T01:00:13.821 DEBUG:teuthology.orchestra.run.smithi090:> sudo rpm -qi grub2-tools 2024-03-18T01:00:13.854 INFO:teuthology.orchestra.run.smithi090.stdout:Name : grub2-tools 2024-03-18T01:00:13.854 INFO:teuthology.orchestra.run.smithi090.stdout:Epoch : 1 2024-03-18T01:00:13.854 INFO:teuthology.orchestra.run.smithi090.stdout:Version : 2.06 2024-03-18T01:00:13.854 INFO:teuthology.orchestra.run.smithi090.stdout:Release : 68.el9 2024-03-18T01:00:13.855 INFO:teuthology.orchestra.run.smithi090.stdout:Architecture: x86_64 2024-03-18T01:00:13.855 INFO:teuthology.orchestra.run.smithi090.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-18T01:00:13.855 INFO:teuthology.orchestra.run.smithi090.stdout:Group : Unspecified 2024-03-18T01:00:13.855 INFO:teuthology.orchestra.run.smithi090.stdout:Size : 8264254 2024-03-18T01:00:13.855 INFO:teuthology.orchestra.run.smithi090.stdout:License : GPLv3+ 2024-03-18T01:00:13.855 INFO:teuthology.orchestra.run.smithi090.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-18T01:00:13.855 INFO:teuthology.orchestra.run.smithi090.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-18T01:00:13.855 INFO:teuthology.orchestra.run.smithi090.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-18T01:00:13.855 INFO:teuthology.orchestra.run.smithi090.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-18T01:00:13.855 INFO:teuthology.orchestra.run.smithi090.stdout:Packager : builder@centos.org 2024-03-18T01:00:13.855 INFO:teuthology.orchestra.run.smithi090.stdout:Vendor : CentOS 2024-03-18T01:00:13.855 INFO:teuthology.orchestra.run.smithi090.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-18T01:00:13.855 INFO:teuthology.orchestra.run.smithi090.stdout:Summary : Support tools for GRUB. 2024-03-18T01:00:13.855 INFO:teuthology.orchestra.run.smithi090.stdout:Description : 2024-03-18T01:00:13.855 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-18T01:00:13.856 INFO:teuthology.orchestra.run.smithi090.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-18T01:00:13.856 INFO:teuthology.orchestra.run.smithi090.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-18T01:00:13.856 INFO:teuthology.orchestra.run.smithi090.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-18T01:00:13.856 INFO:teuthology.orchestra.run.smithi090.stdout:hardware devices. 2024-03-18T01:00:13.856 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-18T01:00:13.856 INFO:teuthology.orchestra.run.smithi090.stdout:This subpackage provides tools for support of all platforms. 2024-03-18T01:00:13.857 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-18T01:00:13.857 INFO:teuthology.task.kernel:Updating grub on smithi090 to boot 5.14.0-427.el9.x86_64 2024-03-18T01:00:13.857 DEBUG:teuthology.orchestra.run.smithi090:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-18T01:00:14.554 INFO:teuthology.orchestra.run.smithi090.stderr:Generating grub configuration file ... 2024-03-18T01:00:16.471 INFO:teuthology.orchestra.run.smithi090.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-18T01:00:16.491 INFO:teuthology.orchestra.run.smithi090.stderr:done 2024-03-18T01:00:16.494 DEBUG:teuthology.orchestra.run.smithi090:> mktemp 2024-03-18T01:00:16.525 INFO:teuthology.orchestra.run.smithi090.stdout:/tmp/tmp.y2MuhLnGBE 2024-03-18T01:00:16.525 DEBUG:teuthology.orchestra.run.smithi090:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.y2MuhLnGBE 2024-03-18T01:00:16.592 DEBUG:teuthology.orchestra.run.smithi090:> sudo chmod 0666 /tmp/tmp.y2MuhLnGBE 2024-03-18T01:00:16.740 DEBUG:teuthology.orchestra.remote:smithi090:/tmp/tmp.y2MuhLnGBE is 6KB 2024-03-18T01:00:16.788 DEBUG:teuthology.orchestra.run.smithi090:> rm -fr /tmp/tmp.y2MuhLnGBE 2024-03-18T01:00:16.802 DEBUG:teuthology.orchestra.run.smithi090:> sudo /bin/ls /boot/loader/entries || true 2024-03-18T01:00:16.866 INFO:teuthology.orchestra.run.smithi090.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-18T01:00:16.866 INFO:teuthology.orchestra.run.smithi090.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-18T01:00:16.866 INFO:teuthology.orchestra.run.smithi090.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-18T01:00:16.866 INFO:teuthology.orchestra.run.smithi090.stdout:15949e3c88704300882482eddfd7baae-5.14.0-427.el9.x86_64.conf 2024-03-18T01:00:16.867 DEBUG:teuthology.orchestra.run.smithi090:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-427.el9.x86_64 2024-03-18T01:00:17.005 DEBUG:teuthology.orchestra.run.smithi090:> sudo shutdown -r now 2024-03-18T01:00:24.082 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: kernel-core-5.14.0-427.el9.x86_64 4/4 2024-03-18T01:00:24.082 INFO:teuthology.orchestra.run.smithi029.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-18T01:00:24.082 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-18T01:00:24.168 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: kernel-modules-5.14.0-427.el9.x86_64 4/4 2024-03-18T01:00:24.687 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: kernel-5.14.0-427.el9.x86_64 4/4 2024-03-18T01:00:24.688 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : kernel-5.14.0-427.el9.x86_64 1/4 2024-03-18T01:00:24.688 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-18T01:00:24.688 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-18T01:00:25.178 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : kernel-modules-core-5.14.0-427.el9.x86_64 4/4 2024-03-18T01:00:25.178 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-18T01:00:25.178 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2024-03-18T01:00:25.178 INFO:teuthology.orchestra.run.smithi029.stdout: kernel-5.14.0-427.el9.x86_64 2024-03-18T01:00:25.178 INFO:teuthology.orchestra.run.smithi029.stdout: kernel-core-5.14.0-427.el9.x86_64 2024-03-18T01:00:25.178 INFO:teuthology.orchestra.run.smithi029.stdout: kernel-modules-5.14.0-427.el9.x86_64 2024-03-18T01:00:25.178 INFO:teuthology.orchestra.run.smithi029.stdout: kernel-modules-core-5.14.0-427.el9.x86_64 2024-03-18T01:00:25.178 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-18T01:00:25.179 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2024-03-18T01:00:25.909 DEBUG:teuthology.orchestra.run.smithi029:> rpm -q kernel | sort -rV | head -n 1 2024-03-18T01:00:26.078 INFO:teuthology.orchestra.run.smithi029.stdout:kernel-5.14.0-427.el9.x86_64 2024-03-18T01:00:26.078 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-427.el9.x86_64 2024-03-18T01:00:26.079 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-427.el9.x86_64 2024-03-18T01:00:26.079 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-18T01:00:26.079 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-18T01:00:26.079 INFO:teuthology.task.kernel:Installing distro kernel on host.a... 2024-03-18T01:00:26.079 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi029.front.sepia.ceph.com, path=None, version=distro) 2024-03-18T01:00:26.079 DEBUG:teuthology.orchestra.run.smithi029:> rpm -q kernel | sort -rV | head -n 1 2024-03-18T01:00:26.205 INFO:teuthology.orchestra.run.smithi029.stdout:kernel-5.14.0-427.el9.x86_64 2024-03-18T01:00:26.205 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-427.el9.x86_64 2024-03-18T01:00:26.205 DEBUG:teuthology.orchestra.run.smithi029:> sudo rpm -qi grub2-tools 2024-03-18T01:00:26.242 INFO:teuthology.orchestra.run.smithi029.stdout:Name : grub2-tools 2024-03-18T01:00:26.242 INFO:teuthology.orchestra.run.smithi029.stdout:Epoch : 1 2024-03-18T01:00:26.242 INFO:teuthology.orchestra.run.smithi029.stdout:Version : 2.06 2024-03-18T01:00:26.242 INFO:teuthology.orchestra.run.smithi029.stdout:Release : 68.el9 2024-03-18T01:00:26.243 INFO:teuthology.orchestra.run.smithi029.stdout:Architecture: x86_64 2024-03-18T01:00:26.243 INFO:teuthology.orchestra.run.smithi029.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-18T01:00:26.243 INFO:teuthology.orchestra.run.smithi029.stdout:Group : Unspecified 2024-03-18T01:00:26.243 INFO:teuthology.orchestra.run.smithi029.stdout:Size : 8264254 2024-03-18T01:00:26.243 INFO:teuthology.orchestra.run.smithi029.stdout:License : GPLv3+ 2024-03-18T01:00:26.243 INFO:teuthology.orchestra.run.smithi029.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-18T01:00:26.243 INFO:teuthology.orchestra.run.smithi029.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-18T01:00:26.243 INFO:teuthology.orchestra.run.smithi029.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-18T01:00:26.243 INFO:teuthology.orchestra.run.smithi029.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-18T01:00:26.243 INFO:teuthology.orchestra.run.smithi029.stdout:Packager : builder@centos.org 2024-03-18T01:00:26.243 INFO:teuthology.orchestra.run.smithi029.stdout:Vendor : CentOS 2024-03-18T01:00:26.243 INFO:teuthology.orchestra.run.smithi029.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-18T01:00:26.243 INFO:teuthology.orchestra.run.smithi029.stdout:Summary : Support tools for GRUB. 2024-03-18T01:00:26.243 INFO:teuthology.orchestra.run.smithi029.stdout:Description : 2024-03-18T01:00:26.243 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-18T01:00:26.243 INFO:teuthology.orchestra.run.smithi029.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-18T01:00:26.244 INFO:teuthology.orchestra.run.smithi029.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-18T01:00:26.244 INFO:teuthology.orchestra.run.smithi029.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-18T01:00:26.244 INFO:teuthology.orchestra.run.smithi029.stdout:hardware devices. 2024-03-18T01:00:26.244 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-18T01:00:26.244 INFO:teuthology.orchestra.run.smithi029.stdout:This subpackage provides tools for support of all platforms. 2024-03-18T01:00:26.245 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-18T01:00:26.245 INFO:teuthology.task.kernel:Updating grub on smithi029 to boot 5.14.0-427.el9.x86_64 2024-03-18T01:00:26.245 DEBUG:teuthology.orchestra.run.smithi029:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-18T01:00:26.975 INFO:teuthology.orchestra.run.smithi029.stderr:Generating grub configuration file ... 2024-03-18T01:00:28.586 INFO:teuthology.orchestra.run.smithi029.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-18T01:00:28.611 INFO:teuthology.orchestra.run.smithi029.stderr:done 2024-03-18T01:00:28.614 DEBUG:teuthology.orchestra.run.smithi029:> mktemp 2024-03-18T01:00:28.629 INFO:teuthology.orchestra.run.smithi029.stdout:/tmp/tmp.cvsQEPRB4Q 2024-03-18T01:00:28.629 DEBUG:teuthology.orchestra.run.smithi029:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.cvsQEPRB4Q 2024-03-18T01:00:28.705 DEBUG:teuthology.orchestra.run.smithi029:> sudo chmod 0666 /tmp/tmp.cvsQEPRB4Q 2024-03-18T01:00:28.850 DEBUG:teuthology.orchestra.remote:smithi029:/tmp/tmp.cvsQEPRB4Q is 6KB 2024-03-18T01:00:28.898 DEBUG:teuthology.orchestra.run.smithi029:> rm -fr /tmp/tmp.cvsQEPRB4Q 2024-03-18T01:00:28.913 DEBUG:teuthology.orchestra.run.smithi029:> sudo /bin/ls /boot/loader/entries || true 2024-03-18T01:00:28.985 INFO:teuthology.orchestra.run.smithi029.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-18T01:00:28.985 INFO:teuthology.orchestra.run.smithi029.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-18T01:00:28.986 INFO:teuthology.orchestra.run.smithi029.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-18T01:00:28.986 INFO:teuthology.orchestra.run.smithi029.stdout:15949e3c88704300882482eddfd7baae-5.14.0-427.el9.x86_64.conf 2024-03-18T01:00:28.987 DEBUG:teuthology.orchestra.run.smithi029:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-427.el9.x86_64 2024-03-18T01:00:29.145 DEBUG:teuthology.orchestra.run.smithi029:> sudo shutdown -r now 2024-03-18T01:00:47.026 INFO:teuthology.task.kernel:Checking client host.b for new kernel version... 2024-03-18T01:00:47.027 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2024-03-18T01:00:47.028 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-18T01:00:59.159 INFO:teuthology.task.kernel:Checking client host.a for new kernel version... 2024-03-18T01:00:59.160 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi029.front.sepia.ceph.com' 2024-03-18T01:00:59.160 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi029.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-18T01:01:05.438 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.90 2024-03-18T01:01:14.446 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2024-03-18T01:01:14.447 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-18T01:01:17.505 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.90 2024-03-18T01:01:17.726 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.29 2024-03-18T01:01:26.734 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi029.front.sepia.ceph.com' 2024-03-18T01:01:26.735 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi029.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-18T01:01:29.509 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2024-03-18T01:01:29.510 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-18T01:01:32.574 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.90 2024-03-18T01:01:45.118 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.29 2024-03-18T01:01:47.576 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2024-03-18T01:01:47.577 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-18T01:01:57.128 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi029.front.sepia.ceph.com' 2024-03-18T01:01:57.129 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi029.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-18T01:02:00.190 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.29 2024-03-18T01:02:06.110 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.90 2024-03-18T01:02:15.198 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi029.front.sepia.ceph.com' 2024-03-18T01:02:15.199 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi029.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-18T01:02:16.120 DEBUG:teuthology.orchestra.run.smithi029:> true 2024-03-18T01:02:16.523 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi029.front.sepia.ceph.com' 2024-03-18T01:02:16.523 INFO:teuthology.task.kernel:Checking kernel version of host.a, want "5.14.0-427.el9.x86_64"... 2024-03-18T01:02:16.523 DEBUG:teuthology.orchestra.run.smithi029:> uname -r 2024-03-18T01:02:16.537 INFO:teuthology.orchestra.run.smithi029.stdout:5.14.0-427.el9.x86_64 2024-03-18T01:02:16.537 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-427.el9.x86_64 vs 5.14.0-427.el9.x86_64 2024-03-18T01:02:16.537 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-18T01:02:16.537 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-18T01:02:17.538 INFO:teuthology.task.kernel:Enabling kdb on host.a... 2024-03-18T01:02:17.538 DEBUG:teuthology.orchestra.run.smithi029:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-18T01:02:17.657 INFO:teuthology.orchestra.run.smithi029.stdout:ttyS1 2024-03-18T01:02:17.696 DEBUG:teuthology.parallel:result is None 2024-03-18T01:02:24.112 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2024-03-18T01:02:24.112 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-18T01:02:24.494 DEBUG:teuthology.orchestra.run.smithi090:> true 2024-03-18T01:02:25.063 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi090.front.sepia.ceph.com' 2024-03-18T01:02:25.063 INFO:teuthology.task.kernel:Checking kernel version of host.b, want "5.14.0-427.el9.x86_64"... 2024-03-18T01:02:25.063 DEBUG:teuthology.orchestra.run.smithi090:> uname -r 2024-03-18T01:02:25.077 INFO:teuthology.orchestra.run.smithi090.stdout:5.14.0-427.el9.x86_64 2024-03-18T01:02:25.077 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-427.el9.x86_64 vs 5.14.0-427.el9.x86_64 2024-03-18T01:02:25.077 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-18T01:02:25.078 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-18T01:02:26.078 INFO:teuthology.task.kernel:Enabling kdb on host.b... 2024-03-18T01:02:26.079 DEBUG:teuthology.orchestra.run.smithi090:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-18T01:02:26.551 INFO:teuthology.orchestra.run.smithi090.stdout:ttyS1 2024-03-18T01:02:26.673 DEBUG:teuthology.parallel:result is None 2024-03-18T01:02:26.673 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-18T01:02:26.680 INFO:teuthology.task.internal:Creating test directory... 2024-03-18T01:02:26.680 DEBUG:teuthology.orchestra.run.smithi029:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-18T01:02:26.683 DEBUG:teuthology.orchestra.run.smithi090:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-18T01:02:26.702 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-18T01:02:26.709 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-18T01:02:26.734 INFO:teuthology.task.internal:Creating archive directory... 2024-03-18T01:02:26.734 DEBUG:teuthology.orchestra.run.smithi029:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-18T01:02:26.741 DEBUG:teuthology.orchestra.run.smithi090:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-18T01:02:26.790 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-18T01:02:26.796 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-18T01:02:26.797 DEBUG:teuthology.orchestra.run.smithi029:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-18T01:02:26.811 DEBUG:teuthology.orchestra.run.smithi090:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-18T01:02:26.844 INFO:teuthology.orchestra.run.smithi029.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-18T01:02:26.858 INFO:teuthology.orchestra.run.smithi029.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-18T01:02:26.879 INFO:teuthology.orchestra.run.smithi090.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-18T01:02:26.898 INFO:teuthology.orchestra.run.smithi090.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-18T01:02:26.901 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-18T01:02:26.907 INFO:teuthology.task.internal:Configuring sudo... 2024-03-18T01:02:26.908 DEBUG:teuthology.orchestra.run.smithi029:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-18T01:02:26.910 DEBUG:teuthology.orchestra.run.smithi090:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-18T01:02:26.952 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-18T01:02:26.961 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-18T01:02:26.962 DEBUG:teuthology.orchestra.run.smithi029:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-18T01:02:26.980 DEBUG:teuthology.orchestra.run.smithi090:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-18T01:02:27.022 DEBUG:teuthology.orchestra.run.smithi029:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-18T01:02:27.076 DEBUG:teuthology.orchestra.run.smithi029:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-18T01:02:27.130 DEBUG:teuthology.orchestra.run.smithi029:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-18T01:02:27.214 DEBUG:teuthology.orchestra.run.smithi029:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-18T01:02:27.254 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-03-18T01:02:27.254 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-18T01:02:27.321 DEBUG:teuthology.orchestra.run.smithi090:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-18T01:02:27.373 DEBUG:teuthology.orchestra.run.smithi090:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-18T01:02:27.441 DEBUG:teuthology.orchestra.run.smithi090:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-18T01:02:27.494 DEBUG:teuthology.orchestra.run.smithi090:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-18T01:02:27.552 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-03-18T01:02:27.552 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-18T01:02:27.635 DEBUG:teuthology.orchestra.run.smithi029:> sudo service rsyslog restart 2024-03-18T01:02:27.638 DEBUG:teuthology.orchestra.run.smithi090:> sudo service rsyslog restart 2024-03-18T01:02:27.683 INFO:teuthology.orchestra.run.smithi029.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-18T01:02:27.748 INFO:teuthology.orchestra.run.smithi090.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-18T01:02:28.104 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-18T01:02:28.111 INFO:teuthology.task.internal:Starting timer... 2024-03-18T01:02:28.111 INFO:teuthology.run_tasks:Running task pcp... 2024-03-18T01:02:28.122 INFO:teuthology.run_tasks:Running task selinux... 2024-03-18T01:02:28.131 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0', 'scontext=system_u:system_r:getty_t:s0']} 2024-03-18T01:02:28.132 DEBUG:teuthology.orchestra.run.smithi029:> sudo service auditd rotate 2024-03-18T01:02:28.202 INFO:teuthology.orchestra.run.smithi029.stdout:Rotating logs: 2024-03-18T01:02:28.204 DEBUG:teuthology.orchestra.run.smithi090:> sudo service auditd rotate 2024-03-18T01:02:28.269 INFO:teuthology.orchestra.run.smithi090.stdout:Rotating logs: 2024-03-18T01:02:28.271 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-18T01:02:28.271 DEBUG:teuthology.orchestra.run.smithi029:> /usr/sbin/getenforce 2024-03-18T01:02:28.295 INFO:teuthology.orchestra.run.smithi029.stdout:Permissive 2024-03-18T01:02:28.296 DEBUG:teuthology.orchestra.run.smithi090:> /usr/sbin/getenforce 2024-03-18T01:02:28.479 INFO:teuthology.orchestra.run.smithi090.stdout:Permissive 2024-03-18T01:02:28.480 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi029.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi090.front.sepia.ceph.com': 'permissive'} 2024-03-18T01:02:28.480 DEBUG:teuthology.orchestra.run.smithi029:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-18T01:02:28.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-18T01:02:28.508 DEBUG:teuthology.orchestra.run.smithi090:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-18T01:02:28.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-18T01:02:28.535 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-18T01:02:28.535 DEBUG:teuthology.orchestra.run.smithi029:> sudo /usr/sbin/setenforce permissive 2024-03-18T01:02:28.582 DEBUG:teuthology.orchestra.run.smithi090:> sudo /usr/sbin/setenforce permissive 2024-03-18T01:02:28.638 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-18T01:02:28.648 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-18T01:02:28.667 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-18T01:02:28.669 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi029.front.sepia.ceph.com,smithi090.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-18T01:06:53.230 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi029.front.sepia.ceph.com'), Remote(name='ubuntu@smithi090.front.sepia.ceph.com')] 2024-03-18T01:06:53.231 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi029.front.sepia.ceph.com' 2024-03-18T01:06:53.232 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi029.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-18T01:06:53.325 DEBUG:teuthology.orchestra.run.smithi029:> true 2024-03-18T01:06:53.402 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi029.front.sepia.ceph.com' 2024-03-18T01:06:53.403 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2024-03-18T01:06:53.404 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-18T01:06:53.495 DEBUG:teuthology.orchestra.run.smithi090:> true 2024-03-18T01:06:53.585 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi090.front.sepia.ceph.com' 2024-03-18T01:06:53.585 INFO:teuthology.run_tasks:Running task clock... 2024-03-18T01:06:53.596 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-18T01:06:53.596 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-18T01:06:53.597 DEBUG:teuthology.orchestra.run.smithi029:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-18T01:06:53.599 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-18T01:06:53.599 DEBUG:teuthology.orchestra.run.smithi090:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-18T01:06:53.628 INFO:teuthology.orchestra.run.smithi029.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-18T01:06:53.646 INFO:teuthology.orchestra.run.smithi029.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-18T01:06:53.657 INFO:teuthology.orchestra.run.smithi090.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-18T01:06:53.674 INFO:teuthology.orchestra.run.smithi090.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-18T01:06:53.675 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: ntpd: command not found 2024-03-18T01:06:53.689 INFO:teuthology.orchestra.run.smithi029.stdout:506 Cannot talk to daemon 2024-03-18T01:06:53.703 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: ntpd: command not found 2024-03-18T01:06:53.705 INFO:teuthology.orchestra.run.smithi029.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-18T01:06:53.717 INFO:teuthology.orchestra.run.smithi090.stdout:506 Cannot talk to daemon 2024-03-18T01:06:53.722 INFO:teuthology.orchestra.run.smithi029.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-18T01:06:53.734 INFO:teuthology.orchestra.run.smithi090.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-18T01:06:53.750 INFO:teuthology.orchestra.run.smithi090.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-18T01:06:53.769 INFO:teuthology.orchestra.run.smithi029.stderr:bash: line 1: ntpq: command not found 2024-03-18T01:06:53.773 INFO:teuthology.orchestra.run.smithi029.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-18T01:06:53.773 INFO:teuthology.orchestra.run.smithi029.stdout:=============================================================================== 2024-03-18T01:06:53.773 INFO:teuthology.orchestra.run.smithi029.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-18T01:06:53.773 INFO:teuthology.orchestra.run.smithi029.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-18T01:06:53.773 INFO:teuthology.orchestra.run.smithi029.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-18T01:06:53.773 INFO:teuthology.orchestra.run.smithi029.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-18T01:06:53.804 INFO:teuthology.orchestra.run.smithi090.stderr:bash: line 1: ntpq: command not found 2024-03-18T01:06:53.807 INFO:teuthology.orchestra.run.smithi090.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-18T01:06:53.807 INFO:teuthology.orchestra.run.smithi090.stdout:=============================================================================== 2024-03-18T01:06:53.807 INFO:teuthology.orchestra.run.smithi090.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-18T01:06:53.808 INFO:teuthology.orchestra.run.smithi090.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-18T01:06:53.808 INFO:teuthology.orchestra.run.smithi090.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-18T01:06:53.808 INFO:teuthology.orchestra.run.smithi090.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-18T01:06:53.809 INFO:teuthology.run_tasks:Running task install... 2024-03-18T01:06:53.817 DEBUG:teuthology.task.install:project ceph 2024-03-18T01:06:53.817 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '3d78f5ba07bf18880172c07d4d7b4ac00c9d30af'}} 2024-03-18T01:06:53.817 DEBUG:teuthology.task.install:config {'branch': 'quincy', 'exclude_packages': ['ceph-volume'], 'flavor': 'default', 'sha1': '3d78f5ba07bf18880172c07d4d7b4ac00c9d30af'} 2024-03-18T01:06:53.817 INFO:teuthology.task.install:Using flavor: default 2024-03-18T01:06:53.823 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-03-18T01:06:53.823 INFO:teuthology.task.install:extra packages: [] 2024-03-18T01:06:53.823 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': 'quincy', 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '3d78f5ba07bf18880172c07d4d7b4ac00c9d30af', 'tag': None, 'wait_for_package': False} 2024-03-18T01:06:53.824 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-18T01:06:53.824 INFO:teuthology.packaging:ref: None 2024-03-18T01:06:53.824 INFO:teuthology.packaging:tag: None 2024-03-18T01:06:53.824 INFO:teuthology.packaging:branch: quincy 2024-03-18T01:06:53.824 INFO:teuthology.packaging:sha1: 3d78f5ba07bf18880172c07d4d7b4ac00c9d30af 2024-03-18T01:06:53.824 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=quincy 2024-03-18T01:06:53.826 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': 'quincy', 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '3d78f5ba07bf18880172c07d4d7b4ac00c9d30af', 'tag': None, 'wait_for_package': False} 2024-03-18T01:06:53.826 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-18T01:06:53.826 INFO:teuthology.packaging:ref: None 2024-03-18T01:06:53.826 INFO:teuthology.packaging:tag: None 2024-03-18T01:06:53.826 INFO:teuthology.packaging:branch: quincy 2024-03-18T01:06:53.826 INFO:teuthology.packaging:sha1: 3d78f5ba07bf18880172c07d4d7b4ac00c9d30af 2024-03-18T01:06:53.826 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=quincy 2024-03-18T01:06:54.054 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/quincy/6716c9f3976cf9fc2fd546e9f1182c2d4421b451/centos/9/flavors/default/ 2024-03-18T01:06:54.055 INFO:teuthology.task.install.rpm:Package version is 17.2.7-684.g6716c9f3 2024-03-18T01:06:54.075 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/quincy/6716c9f3976cf9fc2fd546e9f1182c2d4421b451/centos/9/flavors/default/ 2024-03-18T01:06:54.076 INFO:teuthology.task.install.rpm:Package version is 17.2.7-684.g6716c9f3 2024-03-18T01:06:54.227 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/quincy/6716c9f3976cf9fc2fd546e9f1182c2d4421b451/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/quincy/6716c9f3976cf9fc2fd546e9f1182c2d4421b451/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/quincy/6716c9f3976cf9fc2fd546e9f1182c2d4421b451/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-18T01:06:54.227 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-03-18T01:06:54.228 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-18T01:06:54.246 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/quincy/6716c9f3976cf9fc2fd546e9f1182c2d4421b451/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/quincy/6716c9f3976cf9fc2fd546e9f1182c2d4421b451/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/quincy/6716c9f3976cf9fc2fd546e9f1182c2d4421b451/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-18T01:06:54.246 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-03-18T01:06:54.246 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-18T01:06:54.257 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-03-18T01:06:54.257 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-18T01:06:54.257 INFO:teuthology.packaging:ref: None 2024-03-18T01:06:54.257 INFO:teuthology.packaging:tag: None 2024-03-18T01:06:54.257 INFO:teuthology.packaging:branch: quincy 2024-03-18T01:06:54.257 INFO:teuthology.packaging:sha1: 3d78f5ba07bf18880172c07d4d7b4ac00c9d30af 2024-03-18T01:06:54.257 DEBUG:teuthology.orchestra.run.smithi029:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/quincy/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-18T01:06:54.275 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-03-18T01:06:54.275 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-18T01:06:54.275 INFO:teuthology.packaging:ref: None 2024-03-18T01:06:54.275 INFO:teuthology.packaging:tag: None 2024-03-18T01:06:54.276 INFO:teuthology.packaging:branch: quincy 2024-03-18T01:06:54.276 INFO:teuthology.packaging:sha1: 3d78f5ba07bf18880172c07d4d7b4ac00c9d30af 2024-03-18T01:06:54.276 DEBUG:teuthology.orchestra.run.smithi090:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/quincy/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-18T01:06:54.325 DEBUG:teuthology.orchestra.run.smithi029:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-18T01:06:54.344 DEBUG:teuthology.orchestra.run.smithi090:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-18T01:06:54.405 DEBUG:teuthology.orchestra.run.smithi029:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-18T01:06:54.424 DEBUG:teuthology.orchestra.run.smithi090:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-18T01:06:54.472 INFO:teuthology.orchestra.run.smithi029.stdout:check_obsoletes = 1 2024-03-18T01:06:54.474 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum clean all 2024-03-18T01:06:54.491 INFO:teuthology.orchestra.run.smithi090.stdout:check_obsoletes = 1 2024-03-18T01:06:54.492 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum clean all 2024-03-18T01:06:54.719 INFO:teuthology.orchestra.run.smithi029.stdout:74 files removed 2024-03-18T01:06:54.734 INFO:teuthology.orchestra.run.smithi090.stdout:74 files removed 2024-03-18T01:06:54.746 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install ceph-radosgw 2024-03-18T01:06:54.761 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install ceph-radosgw 2024-03-18T01:06:55.437 INFO:teuthology.orchestra.run.smithi090.stdout:ceph packages for x86_64 215 kB/s | 77 kB 00:00 2024-03-18T01:06:55.443 INFO:teuthology.orchestra.run.smithi029.stdout:ceph packages for x86_64 206 kB/s | 77 kB 00:00 2024-03-18T01:06:55.798 INFO:teuthology.orchestra.run.smithi090.stdout:ceph noarch packages 29 kB/s | 9.2 kB 00:00 2024-03-18T01:06:55.811 INFO:teuthology.orchestra.run.smithi029.stdout:ceph noarch packages 29 kB/s | 9.2 kB 00:00 2024-03-18T01:06:56.179 INFO:teuthology.orchestra.run.smithi029.stdout:ceph source packages 5.5 kB/s | 1.8 kB 00:00 2024-03-18T01:06:56.290 INFO:teuthology.orchestra.run.smithi090.stdout:ceph source packages 4.0 kB/s | 1.8 kB 00:00 2024-03-18T01:06:56.675 INFO:teuthology.orchestra.run.smithi029.stdout:CentOS-9 - AppStream 15 MB/s | 6.7 MB 00:00 2024-03-18T01:06:56.876 INFO:teuthology.orchestra.run.smithi090.stdout:CentOS-9 - AppStream 12 MB/s | 6.7 MB 00:00 2024-03-18T01:06:58.118 INFO:teuthology.orchestra.run.smithi029.stdout:CentOS-9 - BaseOS 6.9 MB/s | 2.0 MB 00:00 2024-03-18T01:06:58.293 INFO:teuthology.orchestra.run.smithi090.stdout:CentOS-9 - BaseOS 7.0 MB/s | 2.0 MB 00:00 2024-03-18T01:06:59.418 INFO:teuthology.orchestra.run.smithi090.stdout:CentOS Stream 9 - BaseOS 10 MB/s | 8.0 MB 00:00 2024-03-18T01:06:59.627 INFO:teuthology.orchestra.run.smithi029.stdout:CentOS Stream 9 - BaseOS 6.8 MB/s | 8.0 MB 00:01 2024-03-18T01:07:15.661 INFO:teuthology.orchestra.run.smithi090.stdout:CentOS Stream 9 - AppStream 1.2 MB/s | 19 MB 00:15 2024-03-18T01:07:16.194 INFO:teuthology.orchestra.run.smithi029.stdout:CentOS Stream 9 - AppStream 1.2 MB/s | 19 MB 00:15 2024-03-18T01:07:19.916 INFO:teuthology.orchestra.run.smithi090.stdout:CentOS Stream 9 - CRB 7.8 MB/s | 6.1 MB 00:00 2024-03-18T01:07:20.290 INFO:teuthology.orchestra.run.smithi029.stdout:CentOS Stream 9 - CRB 13 MB/s | 6.1 MB 00:00 2024-03-18T01:07:21.306 INFO:teuthology.orchestra.run.smithi090.stdout:CentOS Stream 9 - Extras packages 67 kB/s | 16 kB 00:00 2024-03-18T01:07:21.692 INFO:teuthology.orchestra.run.smithi029.stdout:CentOS Stream 9 - Extras packages 83 kB/s | 16 kB 00:00 2024-03-18T01:07:22.419 INFO:teuthology.orchestra.run.smithi090.stdout:Extra Packages for Enterprise Linux 21 MB/s | 21 MB 00:00 2024-03-18T01:07:27.375 INFO:teuthology.orchestra.run.smithi090.stdout:lab-extras 27 kB/s | 1.7 kB 00:00 2024-03-18T01:07:29.970 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-03-18T01:07:29.985 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-03-18T01:07:29.985 INFO:teuthology.orchestra.run.smithi090.stdout: Package Arch Version Repository Size 2024-03-18T01:07:29.986 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-03-18T01:07:29.986 INFO:teuthology.orchestra.run.smithi090.stdout:Installing: 2024-03-18T01:07:29.986 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-radosgw x86_64 2:17.2.7-684.g6716c9f3.el9 ceph 14 M 2024-03-18T01:07:29.986 INFO:teuthology.orchestra.run.smithi090.stdout:Upgrading: 2024-03-18T01:07:29.986 INFO:teuthology.orchestra.run.smithi090.stdout: librados2 x86_64 2:17.2.7-684.g6716c9f3.el9 ceph 3.1 M 2024-03-18T01:07:29.986 INFO:teuthology.orchestra.run.smithi090.stdout: librbd1 x86_64 2:17.2.7-684.g6716c9f3.el9 ceph 2.9 M 2024-03-18T01:07:29.986 INFO:teuthology.orchestra.run.smithi090.stdout:Installing dependencies: 2024-03-18T01:07:29.986 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-base x86_64 2:17.2.7-684.g6716c9f3.el9 ceph 5.0 M 2024-03-18T01:07:29.986 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-common x86_64 2:17.2.7-684.g6716c9f3.el9 ceph 23 M 2024-03-18T01:07:29.986 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-selinux x86_64 2:17.2.7-684.g6716c9f3.el9 ceph 25 k 2024-03-18T01:07:29.986 INFO:teuthology.orchestra.run.smithi090.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-18T01:07:29.986 INFO:teuthology.orchestra.run.smithi090.stdout: libcephfs2 x86_64 2:17.2.7-684.g6716c9f3.el9 ceph 638 k 2024-03-18T01:07:29.987 INFO:teuthology.orchestra.run.smithi090.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-18T01:07:29.987 INFO:teuthology.orchestra.run.smithi090.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-18T01:07:29.987 INFO:teuthology.orchestra.run.smithi090.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-18T01:07:29.987 INFO:teuthology.orchestra.run.smithi090.stdout: libradosstriper1 x86_64 2:17.2.7-684.g6716c9f3.el9 ceph 457 k 2024-03-18T01:07:29.987 INFO:teuthology.orchestra.run.smithi090.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-18T01:07:29.987 INFO:teuthology.orchestra.run.smithi090.stdout: librgw2 x86_64 2:17.2.7-684.g6716c9f3.el9 ceph 4.9 M 2024-03-18T01:07:29.987 INFO:teuthology.orchestra.run.smithi090.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-18T01:07:29.987 INFO:teuthology.orchestra.run.smithi090.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-18T01:07:29.987 INFO:teuthology.orchestra.run.smithi090.stdout: python3-ceph-argparse x86_64 2:17.2.7-684.g6716c9f3.el9 ceph 45 k 2024-03-18T01:07:29.987 INFO:teuthology.orchestra.run.smithi090.stdout: python3-ceph-common x86_64 2:17.2.7-684.g6716c9f3.el9 ceph 117 k 2024-03-18T01:07:29.987 INFO:teuthology.orchestra.run.smithi090.stdout: python3-cephfs x86_64 2:17.2.7-684.g6716c9f3.el9 ceph 156 k 2024-03-18T01:07:29.987 INFO:teuthology.orchestra.run.smithi090.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-18T01:07:29.987 INFO:teuthology.orchestra.run.smithi090.stdout: python3-rados x86_64 2:17.2.7-684.g6716c9f3.el9 ceph 320 k 2024-03-18T01:07:29.987 INFO:teuthology.orchestra.run.smithi090.stdout: python3-rbd x86_64 2:17.2.7-684.g6716c9f3.el9 ceph 293 k 2024-03-18T01:07:29.988 INFO:teuthology.orchestra.run.smithi090.stdout: python3-rgw x86_64 2:17.2.7-684.g6716c9f3.el9 ceph 99 k 2024-03-18T01:07:29.988 INFO:teuthology.orchestra.run.smithi090.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-18T01:07:29.988 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-18T01:07:29.988 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction Summary 2024-03-18T01:07:29.988 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-03-18T01:07:29.988 INFO:teuthology.orchestra.run.smithi090.stdout:Install 22 Packages 2024-03-18T01:07:29.988 INFO:teuthology.orchestra.run.smithi090.stdout:Upgrade 2 Packages 2024-03-18T01:07:29.988 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-18T01:07:29.988 INFO:teuthology.orchestra.run.smithi090.stdout:Total download size: 58 M 2024-03-18T01:07:29.989 INFO:teuthology.orchestra.run.smithi090.stdout:Downloading Packages: 2024-03-18T01:07:31.077 INFO:teuthology.orchestra.run.smithi090.stdout:(1/24): ceph-base-17.2.7-684.g6716c9f3.el9.x86_ 5.6 MB/s | 5.0 MB 00:00 2024-03-18T01:07:31.144 INFO:teuthology.orchestra.run.smithi090.stdout:(2/24): ceph-selinux-17.2.7-684.g6716c9f3.el9.x 378 kB/s | 25 kB 00:00 2024-03-18T01:07:31.294 INFO:teuthology.orchestra.run.smithi090.stdout:(3/24): libcephfs2-17.2.7-684.g6716c9f3.el9.x86 4.2 MB/s | 638 kB 00:00 2024-03-18T01:07:31.387 INFO:teuthology.orchestra.run.smithi090.stdout:(4/24): libradosstriper1-17.2.7-684.g6716c9f3.e 4.9 MB/s | 457 kB 00:00 2024-03-18T01:07:31.787 INFO:teuthology.orchestra.run.smithi090.stdout:(5/24): ceph-common-17.2.7-684.g6716c9f3.el9.x8 14 MB/s | 23 MB 00:01 2024-03-18T01:07:31.829 INFO:teuthology.orchestra.run.smithi090.stdout:(6/24): python3-ceph-argparse-17.2.7-684.g6716c 1.1 MB/s | 45 kB 00:00 2024-03-18T01:07:31.897 INFO:teuthology.orchestra.run.smithi090.stdout:(7/24): python3-ceph-common-17.2.7-684.g6716c9f 1.7 MB/s | 117 kB 00:00 2024-03-18T01:07:31.972 INFO:teuthology.orchestra.run.smithi090.stdout:(8/24): librgw2-17.2.7-684.g6716c9f3.el9.x86_64 8.4 MB/s | 4.9 MB 00:00 2024-03-18T01:07:31.998 INFO:teuthology.orchestra.run.smithi090.stdout:(9/24): python3-cephfs-17.2.7-684.g6716c9f3.el9 1.5 MB/s | 156 kB 00:00 2024-03-18T01:07:32.031 INFO:teuthology.orchestra.run.smithi090.stdout:(10/24): python3-rados-17.2.7-684.g6716c9f3.el9 5.3 MB/s | 320 kB 00:00 2024-03-18T01:07:32.057 INFO:teuthology.orchestra.run.smithi090.stdout:(11/24): python3-rbd-17.2.7-684.g6716c9f3.el9.x 4.9 MB/s | 293 kB 00:00 2024-03-18T01:07:32.082 INFO:teuthology.orchestra.run.smithi090.stdout:(12/24): python3-rgw-17.2.7-684.g6716c9f3.el9.x 1.9 MB/s | 99 kB 00:00 2024-03-18T01:07:32.182 INFO:teuthology.orchestra.run.smithi090.stdout:(13/24): librabbitmq-0.11.0-7.el9.x86_64.rpm 442 kB/s | 44 kB 00:00 2024-03-18T01:07:32.241 INFO:teuthology.orchestra.run.smithi090.stdout:(14/24): libpmemobj-1.12.1-1.el9.x86_64.rpm 864 kB/s | 159 kB 00:00 2024-03-18T01:07:32.391 INFO:teuthology.orchestra.run.smithi090.stdout:(15/24): librdkafka-1.6.1-102.el9.x86_64.rpm 3.1 MB/s | 661 kB 00:00 2024-03-18T01:07:32.417 INFO:teuthology.orchestra.run.smithi090.stdout:(16/24): lttng-ust-2.12.0-6.el9.x86_64.rpm 1.6 MB/s | 282 kB 00:00 2024-03-18T01:07:32.442 INFO:teuthology.orchestra.run.smithi090.stdout:(17/24): python3-prettytable-0.7.2-27.el9.noarc 821 kB/s | 41 kB 00:00 2024-03-18T01:07:32.726 INFO:teuthology.orchestra.run.smithi090.stdout:(18/24): ceph-radosgw-17.2.7-684.g6716c9f3.el9. 5.5 MB/s | 14 MB 00:02 2024-03-18T01:07:32.751 INFO:teuthology.orchestra.run.smithi090.stdout:(19/24): liboath-2.6.7-2.el9.x86_64.rpm 157 kB/s | 49 kB 00:00 2024-03-18T01:07:32.902 INFO:teuthology.orchestra.run.smithi090.stdout:(20/24): gperftools-libs-2.9.1-2.el9.x86_64.rpm 637 kB/s | 309 kB 00:00 2024-03-18T01:07:32.935 INFO:teuthology.orchestra.run.smithi090.stdout:(21/24): libunwind-1.6.2-1.el9.x86_64.rpm 322 kB/s | 67 kB 00:00 2024-03-18T01:07:33.094 INFO:teuthology.orchestra.run.smithi090.stdout:(22/24): thrift-0.15.0-2.el9.x86_64.rpm 4.6 MB/s | 1.6 MB 00:00 2024-03-18T01:07:33.211 INFO:teuthology.orchestra.run.smithi090.stdout:(23/24): librbd1-17.2.7-684.g6716c9f3.el9.x86_6 11 MB/s | 2.9 MB 00:00 2024-03-18T01:07:33.269 INFO:teuthology.orchestra.run.smithi090.stdout:(24/24): librados2-17.2.7-684.g6716c9f3.el9.x86 8.4 MB/s | 3.1 MB 00:00 2024-03-18T01:07:33.273 INFO:teuthology.orchestra.run.smithi090.stdout:-------------------------------------------------------------------------------- 2024-03-18T01:07:33.273 INFO:teuthology.orchestra.run.smithi090.stdout:Total 18 MB/s | 58 MB 00:03 2024-03-18T01:07:33.274 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction check 2024-03-18T01:07:33.304 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction check succeeded. 2024-03-18T01:07:33.304 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction test 2024-03-18T01:07:33.783 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction test succeeded. 2024-03-18T01:07:33.784 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction 2024-03-18T01:07:34.999 INFO:teuthology.orchestra.run.smithi090.stdout: Preparing : 1/1 2024-03-18T01:07:35.344 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/26 2024-03-18T01:07:35.451 INFO:teuthology.orchestra.run.smithi090.stdout: Upgrading : librados2-2:17.2.7-684.g6716c9f3.el9.x86_64 2/26 2024-03-18T01:07:35.540 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: librados2-2:17.2.7-684.g6716c9f3.el9.x86_64 2/26 2024-03-18T01:07:35.619 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-rados-2:17.2.7-684.g6716c9f3.el9.x86_64 3/26 2024-03-18T01:07:35.642 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : libcephfs2-2:17.2.7-684.g6716c9f3.el9.x86_64 4/26 2024-03-18T01:07:35.729 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: libcephfs2-2:17.2.7-684.g6716c9f3.el9.x86_64 4/26 2024-03-18T01:07:35.783 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : thrift-0.15.0-2.el9.x86_64 5/26 2024-03-18T01:07:35.822 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/26 2024-03-18T01:07:36.261 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 7/26 2024-03-18T01:07:36.287 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : librgw2-2:17.2.7-684.g6716c9f3.el9.x86_64 8/26 2024-03-18T01:07:36.349 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: librgw2-2:17.2.7-684.g6716c9f3.el9.x86_64 8/26 2024-03-18T01:07:36.387 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : liboath-2.6.7-2.el9.x86_64 9/26 2024-03-18T01:07:36.429 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-ceph-argparse-2:17.2.7-684.g6716c9f3.el9.x 10/26 2024-03-18T01:07:36.478 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-cephfs-2:17.2.7-684.g6716c9f3.el9.x86_64 11/26 2024-03-18T01:07:36.537 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-rgw-2:17.2.7-684.g6716c9f3.el9.x86_64 12/26 2024-03-18T01:07:36.558 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : libradosstriper1-2:17.2.7-684.g6716c9f3.el9.x86_64 13/26 2024-03-18T01:07:36.623 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: libradosstriper1-2:17.2.7-684.g6716c9f3.el9.x86_64 13/26 2024-03-18T01:07:36.663 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/26 2024-03-18T01:07:36.700 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 15/26 2024-03-18T01:07:36.782 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 16/26 2024-03-18T01:07:37.335 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 17/26 2024-03-18T01:07:37.363 INFO:teuthology.orchestra.run.smithi090.stdout: Upgrading : librbd1-2:17.2.7-684.g6716c9f3.el9.x86_64 18/26 2024-03-18T01:07:37.452 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: librbd1-2:17.2.7-684.g6716c9f3.el9.x86_64 18/26 2024-03-18T01:07:37.525 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-rbd-2:17.2.7-684.g6716c9f3.el9.x86_64 19/26 2024-03-18T01:07:37.557 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-ceph-common-2:17.2.7-684.g6716c9f3.el9.x86 20/26 2024-03-18T01:07:40.096 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-common-2:17.2.7-684.g6716c9f3.el9.x86_64 21/26 2024-03-18T01:07:40.156 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-common-2:17.2.7-684.g6716c9f3.el9.x86_64 21/26 2024-03-18T01:07:40.731 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-common-2:17.2.7-684.g6716c9f3.el9.x86_64 21/26 2024-03-18T01:07:40.778 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-base-2:17.2.7-684.g6716c9f3.el9.x86_64 22/26 2024-03-18T01:07:40.868 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-base-2:17.2.7-684.g6716c9f3.el9.x86_64 22/26 2024-03-18T01:07:40.868 INFO:teuthology.orchestra.run.smithi090.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-18T01:07:40.868 INFO:teuthology.orchestra.run.smithi090.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-18T01:07:40.868 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-18T01:07:40.904 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-selinux-2:17.2.7-684.g6716c9f3.el9.x86_64 23/26 2024-03-18T01:07:41.427 INFO:teuthology.orchestra.run.smithi029.stdout:Extra Packages for Enterprise Linux 1.1 MB/s | 21 MB 00:19 2024-03-18T01:07:46.521 INFO:teuthology.orchestra.run.smithi029.stdout:lab-extras 42 kB/s | 1.7 kB 00:00 2024-03-18T01:07:49.236 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2024-03-18T01:07:49.259 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-03-18T01:07:49.259 INFO:teuthology.orchestra.run.smithi029.stdout: Package Arch Version Repository Size 2024-03-18T01:07:49.259 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-03-18T01:07:49.259 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2024-03-18T01:07:49.259 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-radosgw x86_64 2:17.2.7-684.g6716c9f3.el9 ceph 14 M 2024-03-18T01:07:49.260 INFO:teuthology.orchestra.run.smithi029.stdout:Upgrading: 2024-03-18T01:07:49.260 INFO:teuthology.orchestra.run.smithi029.stdout: librados2 x86_64 2:17.2.7-684.g6716c9f3.el9 ceph 3.1 M 2024-03-18T01:07:49.260 INFO:teuthology.orchestra.run.smithi029.stdout: librbd1 x86_64 2:17.2.7-684.g6716c9f3.el9 ceph 2.9 M 2024-03-18T01:07:49.260 INFO:teuthology.orchestra.run.smithi029.stdout:Installing dependencies: 2024-03-18T01:07:49.260 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-base x86_64 2:17.2.7-684.g6716c9f3.el9 ceph 5.0 M 2024-03-18T01:07:49.260 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-common x86_64 2:17.2.7-684.g6716c9f3.el9 ceph 23 M 2024-03-18T01:07:49.260 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-selinux x86_64 2:17.2.7-684.g6716c9f3.el9 ceph 25 k 2024-03-18T01:07:49.260 INFO:teuthology.orchestra.run.smithi029.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-18T01:07:49.260 INFO:teuthology.orchestra.run.smithi029.stdout: libcephfs2 x86_64 2:17.2.7-684.g6716c9f3.el9 ceph 638 k 2024-03-18T01:07:49.260 INFO:teuthology.orchestra.run.smithi029.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-18T01:07:49.260 INFO:teuthology.orchestra.run.smithi029.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-18T01:07:49.260 INFO:teuthology.orchestra.run.smithi029.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-18T01:07:49.260 INFO:teuthology.orchestra.run.smithi029.stdout: libradosstriper1 x86_64 2:17.2.7-684.g6716c9f3.el9 ceph 457 k 2024-03-18T01:07:49.261 INFO:teuthology.orchestra.run.smithi029.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-18T01:07:49.261 INFO:teuthology.orchestra.run.smithi029.stdout: librgw2 x86_64 2:17.2.7-684.g6716c9f3.el9 ceph 4.9 M 2024-03-18T01:07:49.261 INFO:teuthology.orchestra.run.smithi029.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-18T01:07:49.261 INFO:teuthology.orchestra.run.smithi029.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-18T01:07:49.261 INFO:teuthology.orchestra.run.smithi029.stdout: python3-ceph-argparse x86_64 2:17.2.7-684.g6716c9f3.el9 ceph 45 k 2024-03-18T01:07:49.261 INFO:teuthology.orchestra.run.smithi029.stdout: python3-ceph-common x86_64 2:17.2.7-684.g6716c9f3.el9 ceph 117 k 2024-03-18T01:07:49.261 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cephfs x86_64 2:17.2.7-684.g6716c9f3.el9 ceph 156 k 2024-03-18T01:07:49.261 INFO:teuthology.orchestra.run.smithi029.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-18T01:07:49.261 INFO:teuthology.orchestra.run.smithi029.stdout: python3-rados x86_64 2:17.2.7-684.g6716c9f3.el9 ceph 320 k 2024-03-18T01:07:49.261 INFO:teuthology.orchestra.run.smithi029.stdout: python3-rbd x86_64 2:17.2.7-684.g6716c9f3.el9 ceph 293 k 2024-03-18T01:07:49.261 INFO:teuthology.orchestra.run.smithi029.stdout: python3-rgw x86_64 2:17.2.7-684.g6716c9f3.el9 ceph 99 k 2024-03-18T01:07:49.261 INFO:teuthology.orchestra.run.smithi029.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-18T01:07:49.261 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-18T01:07:49.262 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2024-03-18T01:07:49.262 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-03-18T01:07:49.262 INFO:teuthology.orchestra.run.smithi029.stdout:Install 22 Packages 2024-03-18T01:07:49.262 INFO:teuthology.orchestra.run.smithi029.stdout:Upgrade 2 Packages 2024-03-18T01:07:49.262 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-18T01:07:49.262 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 58 M 2024-03-18T01:07:49.263 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2024-03-18T01:07:50.107 INFO:teuthology.orchestra.run.smithi029.stdout:(1/24): ceph-base-17.2.7-684.g6716c9f3.el9.x86_ 6.7 MB/s | 5.0 MB 00:00 2024-03-18T01:07:50.167 INFO:teuthology.orchestra.run.smithi029.stdout:(2/24): ceph-selinux-17.2.7-684.g6716c9f3.el9.x 430 kB/s | 25 kB 00:00 2024-03-18T01:07:50.284 INFO:teuthology.orchestra.run.smithi029.stdout:(3/24): libcephfs2-17.2.7-684.g6716c9f3.el9.x86 5.3 MB/s | 638 kB 00:00 2024-03-18T01:07:50.376 INFO:teuthology.orchestra.run.smithi029.stdout:(4/24): libradosstriper1-17.2.7-684.g6716c9f3.e 4.9 MB/s | 457 kB 00:00 2024-03-18T01:07:50.926 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-selinux-2:17.2.7-684.g6716c9f3.el9.x86_64 23/26 2024-03-18T01:07:50.927 INFO:teuthology.orchestra.run.smithi090.stdout:skipping the directory /sys 2024-03-18T01:07:50.927 INFO:teuthology.orchestra.run.smithi090.stdout:skipping the directory /proc 2024-03-18T01:07:50.927 INFO:teuthology.orchestra.run.smithi090.stdout:skipping the directory /mnt 2024-03-18T01:07:50.927 INFO:teuthology.orchestra.run.smithi090.stdout:skipping the directory /var/tmp 2024-03-18T01:07:50.927 INFO:teuthology.orchestra.run.smithi090.stdout:skipping the directory /home 2024-03-18T01:07:50.927 INFO:teuthology.orchestra.run.smithi090.stdout:skipping the directory /root 2024-03-18T01:07:50.927 INFO:teuthology.orchestra.run.smithi090.stdout:skipping the directory /tmp 2024-03-18T01:07:50.927 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-18T01:07:51.060 INFO:teuthology.orchestra.run.smithi029.stdout:(5/24): ceph-radosgw-17.2.7-684.g6716c9f3.el9.x 8.3 MB/s | 14 MB 00:01 2024-03-18T01:07:51.111 INFO:teuthology.orchestra.run.smithi029.stdout:(6/24): python3-ceph-argparse-17.2.7-684.g6716c 902 kB/s | 45 kB 00:00 2024-03-18T01:07:51.196 INFO:teuthology.orchestra.run.smithi029.stdout:(7/24): python3-ceph-common-17.2.7-684.g6716c9f 1.3 MB/s | 117 kB 00:00 2024-03-18T01:07:51.305 INFO:teuthology.orchestra.run.smithi029.stdout:(8/24): librgw2-17.2.7-684.g6716c9f3.el9.x86_64 5.3 MB/s | 4.9 MB 00:00 2024-03-18T01:07:51.330 INFO:teuthology.orchestra.run.smithi029.stdout:(9/24): python3-cephfs-17.2.7-684.g6716c9f3.el9 1.1 MB/s | 156 kB 00:00 2024-03-18T01:07:51.372 INFO:teuthology.orchestra.run.smithi029.stdout:(10/24): python3-rados-17.2.7-684.g6716c9f3.el9 4.6 MB/s | 320 kB 00:00 2024-03-18T01:07:51.406 INFO:teuthology.orchestra.run.smithi029.stdout:(11/24): python3-rbd-17.2.7-684.g6716c9f3.el9.x 3.8 MB/s | 293 kB 00:00 2024-03-18T01:07:51.431 INFO:teuthology.orchestra.run.smithi029.stdout:(12/24): python3-rgw-17.2.7-684.g6716c9f3.el9.x 1.6 MB/s | 99 kB 00:00 2024-03-18T01:07:51.557 INFO:teuthology.orchestra.run.smithi029.stdout:(13/24): librabbitmq-0.11.0-7.el9.x86_64.rpm 451 kB/s | 44 kB 00:00 2024-03-18T01:07:51.583 INFO:teuthology.orchestra.run.smithi029.stdout:(14/24): libpmemobj-1.12.1-1.el9.x86_64.rpm 903 kB/s | 159 kB 00:00 2024-03-18T01:07:51.642 INFO:teuthology.orchestra.run.smithi029.stdout:(15/24): librdkafka-1.6.1-102.el9.x86_64.rpm 7.7 MB/s | 661 kB 00:00 2024-03-18T01:07:51.675 INFO:teuthology.orchestra.run.smithi029.stdout:(16/24): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.0 MB/s | 282 kB 00:00 2024-03-18T01:07:51.709 INFO:teuthology.orchestra.run.smithi029.stdout:(17/24): python3-prettytable-0.7.2-27.el9.noarc 615 kB/s | 41 kB 00:00 2024-03-18T01:07:51.785 INFO:teuthology.orchestra.run.smithi029.stdout:(18/24): liboath-2.6.7-2.el9.x86_64.rpm 647 kB/s | 49 kB 00:00 2024-03-18T01:07:51.819 INFO:teuthology.orchestra.run.smithi029.stdout:(19/24): gperftools-libs-2.9.1-2.el9.x86_64.rpm 2.1 MB/s | 309 kB 00:00 2024-03-18T01:07:51.844 INFO:teuthology.orchestra.run.smithi029.stdout:(20/24): libunwind-1.6.2-1.el9.x86_64.rpm 1.1 MB/s | 67 kB 00:00 2024-03-18T01:07:51.920 INFO:teuthology.orchestra.run.smithi029.stdout:(21/24): thrift-0.15.0-2.el9.x86_64.rpm 16 MB/s | 1.6 MB 00:00 2024-03-18T01:07:52.240 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-radosgw-2:17.2.7-684.g6716c9f3.el9.x86_64 24/26 2024-03-18T01:07:52.280 INFO:teuthology.orchestra.run.smithi029.stdout:(22/24): librados2-17.2.7-684.g6716c9f3.el9.x86 7.1 MB/s | 3.1 MB 00:00 2024-03-18T01:07:52.318 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-radosgw-2:17.2.7-684.g6716c9f3.el9.x86_64 24/26 2024-03-18T01:07:52.318 INFO:teuthology.orchestra.run.smithi090.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-18T01:07:52.319 INFO:teuthology.orchestra.run.smithi090.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-18T01:07:52.319 INFO:teuthology.orchestra.run.smithi090.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-18T01:07:52.319 INFO:teuthology.orchestra.run.smithi090.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-18T01:07:52.319 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-18T01:07:52.320 INFO:teuthology.orchestra.run.smithi090.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 25/26 2024-03-18T01:07:52.364 INFO:teuthology.orchestra.run.smithi029.stdout:(23/24): librbd1-17.2.7-684.g6716c9f3.el9.x86_6 6.6 MB/s | 2.9 MB 00:00 2024-03-18T01:07:52.411 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 25/26 2024-03-18T01:07:52.411 INFO:teuthology.orchestra.run.smithi090.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 26/26 2024-03-18T01:07:52.840 INFO:teuthology.orchestra.run.smithi029.stdout:(24/24): ceph-common-17.2.7-684.g6716c9f3.el9.x 6.5 MB/s | 23 MB 00:03 2024-03-18T01:07:52.845 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2024-03-18T01:07:52.845 INFO:teuthology.orchestra.run.smithi029.stdout:Total 16 MB/s | 58 MB 00:03 2024-03-18T01:07:52.845 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2024-03-18T01:07:52.882 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2024-03-18T01:07:52.882 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2024-03-18T01:07:53.362 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2024-03-18T01:07:53.362 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2024-03-18T01:07:53.839 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 26/26 2024-03-18T01:07:53.840 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-base-2:17.2.7-684.g6716c9f3.el9.x86_64 1/26 2024-03-18T01:07:53.840 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-common-2:17.2.7-684.g6716c9f3.el9.x86_64 2/26 2024-03-18T01:07:53.840 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-radosgw-2:17.2.7-684.g6716c9f3.el9.x86_64 3/26 2024-03-18T01:07:53.840 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-selinux-2:17.2.7-684.g6716c9f3.el9.x86_64 4/26 2024-03-18T01:07:53.840 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : libcephfs2-2:17.2.7-684.g6716c9f3.el9.x86_64 5/26 2024-03-18T01:07:53.840 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : libradosstriper1-2:17.2.7-684.g6716c9f3.el9.x86_64 6/26 2024-03-18T01:07:53.840 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : librgw2-2:17.2.7-684.g6716c9f3.el9.x86_64 7/26 2024-03-18T01:07:53.840 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-ceph-argparse-2:17.2.7-684.g6716c9f3.el9.x 8/26 2024-03-18T01:07:53.840 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-ceph-common-2:17.2.7-684.g6716c9f3.el9.x86 9/26 2024-03-18T01:07:53.840 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-cephfs-2:17.2.7-684.g6716c9f3.el9.x86_64 10/26 2024-03-18T01:07:53.840 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-rados-2:17.2.7-684.g6716c9f3.el9.x86_64 11/26 2024-03-18T01:07:53.840 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-rbd-2:17.2.7-684.g6716c9f3.el9.x86_64 12/26 2024-03-18T01:07:53.840 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-rgw-2:17.2.7-684.g6716c9f3.el9.x86_64 13/26 2024-03-18T01:07:53.841 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 14/26 2024-03-18T01:07:53.841 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 15/26 2024-03-18T01:07:53.841 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 16/26 2024-03-18T01:07:53.841 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 17/26 2024-03-18T01:07:53.841 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 18/26 2024-03-18T01:07:53.841 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 19/26 2024-03-18T01:07:53.841 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 20/26 2024-03-18T01:07:53.841 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 21/26 2024-03-18T01:07:53.841 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 22/26 2024-03-18T01:07:53.841 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : librados2-2:17.2.7-684.g6716c9f3.el9.x86_64 23/26 2024-03-18T01:07:53.841 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 24/26 2024-03-18T01:07:53.841 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : librbd1-2:17.2.7-684.g6716c9f3.el9.x86_64 25/26 2024-03-18T01:07:54.593 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2024-03-18T01:07:54.804 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 26/26 2024-03-18T01:07:54.804 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-18T01:07:54.804 INFO:teuthology.orchestra.run.smithi090.stdout:Upgraded: 2024-03-18T01:07:54.804 INFO:teuthology.orchestra.run.smithi090.stdout: librados2-2:17.2.7-684.g6716c9f3.el9.x86_64 2024-03-18T01:07:54.804 INFO:teuthology.orchestra.run.smithi090.stdout: librbd1-2:17.2.7-684.g6716c9f3.el9.x86_64 2024-03-18T01:07:54.804 INFO:teuthology.orchestra.run.smithi090.stdout:Installed: 2024-03-18T01:07:54.804 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-base-2:17.2.7-684.g6716c9f3.el9.x86_64 2024-03-18T01:07:54.804 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-common-2:17.2.7-684.g6716c9f3.el9.x86_64 2024-03-18T01:07:54.804 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-radosgw-2:17.2.7-684.g6716c9f3.el9.x86_64 2024-03-18T01:07:54.805 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-selinux-2:17.2.7-684.g6716c9f3.el9.x86_64 2024-03-18T01:07:54.805 INFO:teuthology.orchestra.run.smithi090.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-18T01:07:54.805 INFO:teuthology.orchestra.run.smithi090.stdout: libcephfs2-2:17.2.7-684.g6716c9f3.el9.x86_64 2024-03-18T01:07:54.805 INFO:teuthology.orchestra.run.smithi090.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-18T01:07:54.805 INFO:teuthology.orchestra.run.smithi090.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-18T01:07:54.805 INFO:teuthology.orchestra.run.smithi090.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-18T01:07:54.805 INFO:teuthology.orchestra.run.smithi090.stdout: libradosstriper1-2:17.2.7-684.g6716c9f3.el9.x86_64 2024-03-18T01:07:54.805 INFO:teuthology.orchestra.run.smithi090.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-18T01:07:54.805 INFO:teuthology.orchestra.run.smithi090.stdout: librgw2-2:17.2.7-684.g6716c9f3.el9.x86_64 2024-03-18T01:07:54.805 INFO:teuthology.orchestra.run.smithi090.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-18T01:07:54.805 INFO:teuthology.orchestra.run.smithi090.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-18T01:07:54.805 INFO:teuthology.orchestra.run.smithi090.stdout: python3-ceph-argparse-2:17.2.7-684.g6716c9f3.el9.x86_64 2024-03-18T01:07:54.806 INFO:teuthology.orchestra.run.smithi090.stdout: python3-ceph-common-2:17.2.7-684.g6716c9f3.el9.x86_64 2024-03-18T01:07:54.806 INFO:teuthology.orchestra.run.smithi090.stdout: python3-cephfs-2:17.2.7-684.g6716c9f3.el9.x86_64 2024-03-18T01:07:54.806 INFO:teuthology.orchestra.run.smithi090.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-18T01:07:54.806 INFO:teuthology.orchestra.run.smithi090.stdout: python3-rados-2:17.2.7-684.g6716c9f3.el9.x86_64 2024-03-18T01:07:54.806 INFO:teuthology.orchestra.run.smithi090.stdout: python3-rbd-2:17.2.7-684.g6716c9f3.el9.x86_64 2024-03-18T01:07:54.806 INFO:teuthology.orchestra.run.smithi090.stdout: python3-rgw-2:17.2.7-684.g6716c9f3.el9.x86_64 2024-03-18T01:07:54.806 INFO:teuthology.orchestra.run.smithi090.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-18T01:07:54.806 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-18T01:07:54.806 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-03-18T01:07:54.973 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/26 2024-03-18T01:07:55.094 INFO:teuthology.orchestra.run.smithi029.stdout: Upgrading : librados2-2:17.2.7-684.g6716c9f3.el9.x86_64 2/26 2024-03-18T01:07:55.187 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: librados2-2:17.2.7-684.g6716c9f3.el9.x86_64 2/26 2024-03-18T01:07:55.245 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install ceph-test 2024-03-18T01:07:55.277 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-rados-2:17.2.7-684.g6716c9f3.el9.x86_64 3/26 2024-03-18T01:07:55.307 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : libcephfs2-2:17.2.7-684.g6716c9f3.el9.x86_64 4/26 2024-03-18T01:07:55.403 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: libcephfs2-2:17.2.7-684.g6716c9f3.el9.x86_64 4/26 2024-03-18T01:07:55.454 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : thrift-0.15.0-2.el9.x86_64 5/26 2024-03-18T01:07:55.488 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/26 2024-03-18T01:07:55.806 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:00:28 ago on Mon 18 Mar 2024 01:07:27 AM UTC. 2024-03-18T01:07:55.944 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 7/26 2024-03-18T01:07:55.979 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : librgw2-2:17.2.7-684.g6716c9f3.el9.x86_64 8/26 2024-03-18T01:07:55.984 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-03-18T01:07:55.985 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-03-18T01:07:55.985 INFO:teuthology.orchestra.run.smithi090.stdout: Package Arch Version Repository Size 2024-03-18T01:07:55.985 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-03-18T01:07:55.985 INFO:teuthology.orchestra.run.smithi090.stdout:Installing: 2024-03-18T01:07:55.985 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-test x86_64 2:17.2.7-684.g6716c9f3.el9 ceph 48 M 2024-03-18T01:07:55.985 INFO:teuthology.orchestra.run.smithi090.stdout:Installing dependencies: 2024-03-18T01:07:55.985 INFO:teuthology.orchestra.run.smithi090.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-18T01:07:55.985 INFO:teuthology.orchestra.run.smithi090.stdout: libcephsqlite x86_64 2:17.2.7-684.g6716c9f3.el9 ceph 163 k 2024-03-18T01:07:55.985 INFO:teuthology.orchestra.run.smithi090.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-18T01:07:55.985 INFO:teuthology.orchestra.run.smithi090.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-18T01:07:55.985 INFO:teuthology.orchestra.run.smithi090.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-18T01:07:55.985 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-18T01:07:55.986 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction Summary 2024-03-18T01:07:55.986 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-03-18T01:07:55.986 INFO:teuthology.orchestra.run.smithi090.stdout:Install 6 Packages 2024-03-18T01:07:55.986 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-18T01:07:55.986 INFO:teuthology.orchestra.run.smithi090.stdout:Total download size: 49 M 2024-03-18T01:07:55.986 INFO:teuthology.orchestra.run.smithi090.stdout:Installed size: 196 M 2024-03-18T01:07:55.987 INFO:teuthology.orchestra.run.smithi090.stdout:Downloading Packages: 2024-03-18T01:07:56.042 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: librgw2-2:17.2.7-684.g6716c9f3.el9.x86_64 8/26 2024-03-18T01:07:56.082 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : liboath-2.6.7-2.el9.x86_64 9/26 2024-03-18T01:07:56.125 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-ceph-argparse-2:17.2.7-684.g6716c9f3.el9.x 10/26 2024-03-18T01:07:56.174 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-cephfs-2:17.2.7-684.g6716c9f3.el9.x86_64 11/26 2024-03-18T01:07:56.222 INFO:teuthology.orchestra.run.smithi090.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.2 MB/s | 217 kB 00:00 2024-03-18T01:07:56.243 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-rgw-2:17.2.7-684.g6716c9f3.el9.x86_64 12/26 2024-03-18T01:07:56.269 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : libradosstriper1-2:17.2.7-684.g6716c9f3.el9.x86_64 13/26 2024-03-18T01:07:56.280 INFO:teuthology.orchestra.run.smithi090.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 5.1 MB/s | 300 kB 00:00 2024-03-18T01:07:56.322 INFO:teuthology.orchestra.run.smithi090.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.5 MB/s | 63 kB 00:00 2024-03-18T01:07:56.343 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: libradosstriper1-2:17.2.7-684.g6716c9f3.el9.x86_64 13/26 2024-03-18T01:07:56.355 INFO:teuthology.orchestra.run.smithi090.stdout:(4/6): libcephsqlite-17.2.7-684.g6716c9f3.el9.x 533 kB/s | 163 kB 00:00 2024-03-18T01:07:56.383 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/26 2024-03-18T01:07:56.421 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 15/26 2024-03-18T01:07:56.455 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 16/26 2024-03-18T01:07:56.473 INFO:teuthology.orchestra.run.smithi090.stdout:(5/6): jq-1.6-16.el9.x86_64.rpm 1.2 MB/s | 187 kB 00:00 2024-03-18T01:07:56.764 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 17/26 2024-03-18T01:07:56.795 INFO:teuthology.orchestra.run.smithi029.stdout: Upgrading : librbd1-2:17.2.7-684.g6716c9f3.el9.x86_64 18/26 2024-03-18T01:07:56.886 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: librbd1-2:17.2.7-684.g6716c9f3.el9.x86_64 18/26 2024-03-18T01:07:56.970 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-rbd-2:17.2.7-684.g6716c9f3.el9.x86_64 19/26 2024-03-18T01:07:56.999 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-ceph-common-2:17.2.7-684.g6716c9f3.el9.x86 20/26 2024-03-18T01:07:58.706 INFO:teuthology.orchestra.run.smithi090.stdout:(6/6): ceph-test-17.2.7-684.g6716c9f3.el9.x86_6 18 MB/s | 48 MB 00:02 2024-03-18T01:07:58.710 INFO:teuthology.orchestra.run.smithi090.stdout:-------------------------------------------------------------------------------- 2024-03-18T01:07:58.710 INFO:teuthology.orchestra.run.smithi090.stdout:Total 18 MB/s | 49 MB 00:02 2024-03-18T01:07:58.724 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction check 2024-03-18T01:07:58.745 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction check succeeded. 2024-03-18T01:07:58.745 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction test 2024-03-18T01:07:59.130 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction test succeeded. 2024-03-18T01:07:59.131 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction 2024-03-18T01:07:59.481 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-common-2:17.2.7-684.g6716c9f3.el9.x86_64 21/26 2024-03-18T01:07:59.535 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-common-2:17.2.7-684.g6716c9f3.el9.x86_64 21/26 2024-03-18T01:07:59.728 INFO:teuthology.orchestra.run.smithi090.stdout: Preparing : 1/1 2024-03-18T01:07:59.807 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-18T01:07:59.858 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-18T01:07:59.901 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-18T01:07:59.949 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-18T01:07:59.977 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : libcephsqlite-2:17.2.7-684.g6716c9f3.el9.x86_64 5/6 2024-03-18T01:08:00.108 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-common-2:17.2.7-684.g6716c9f3.el9.x86_64 21/26 2024-03-18T01:08:00.160 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-base-2:17.2.7-684.g6716c9f3.el9.x86_64 22/26 2024-03-18T01:08:00.253 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-base-2:17.2.7-684.g6716c9f3.el9.x86_64 22/26 2024-03-18T01:08:00.253 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-18T01:08:00.254 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-18T01:08:00.254 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-18T01:08:00.348 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-selinux-2:17.2.7-684.g6716c9f3.el9.x86_64 23/26 2024-03-18T01:08:04.423 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: libcephsqlite-2:17.2.7-684.g6716c9f3.el9.x86_64 5/6 2024-03-18T01:08:04.511 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-test-2:17.2.7-684.g6716c9f3.el9.x86_64 6/6 2024-03-18T01:08:07.391 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-test-2:17.2.7-684.g6716c9f3.el9.x86_64 6/6 2024-03-18T01:08:07.391 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-test-2:17.2.7-684.g6716c9f3.el9.x86_64 1/6 2024-03-18T01:08:07.391 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : libcephsqlite-2:17.2.7-684.g6716c9f3.el9.x86_64 2/6 2024-03-18T01:08:07.391 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-18T01:08:07.391 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-18T01:08:07.391 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-18T01:08:07.753 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-18T01:08:07.754 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-18T01:08:07.754 INFO:teuthology.orchestra.run.smithi090.stdout:Installed: 2024-03-18T01:08:07.754 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-test-2:17.2.7-684.g6716c9f3.el9.x86_64 2024-03-18T01:08:07.754 INFO:teuthology.orchestra.run.smithi090.stdout: jq-1.6-16.el9.x86_64 2024-03-18T01:08:07.754 INFO:teuthology.orchestra.run.smithi090.stdout: libcephsqlite-2:17.2.7-684.g6716c9f3.el9.x86_64 2024-03-18T01:08:07.754 INFO:teuthology.orchestra.run.smithi090.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-18T01:08:07.754 INFO:teuthology.orchestra.run.smithi090.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-18T01:08:07.754 INFO:teuthology.orchestra.run.smithi090.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-18T01:08:07.754 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-18T01:08:07.754 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-03-18T01:08:08.114 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install ceph 2024-03-18T01:08:08.681 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:00:41 ago on Mon 18 Mar 2024 01:07:27 AM UTC. 2024-03-18T01:08:08.838 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-03-18T01:08:08.841 INFO:teuthology.orchestra.run.smithi090.stdout:======================================================================================== 2024-03-18T01:08:08.841 INFO:teuthology.orchestra.run.smithi090.stdout: Package Arch Version Repository Size 2024-03-18T01:08:08.841 INFO:teuthology.orchestra.run.smithi090.stdout:======================================================================================== 2024-03-18T01:08:08.841 INFO:teuthology.orchestra.run.smithi090.stdout:Installing: 2024-03-18T01:08:08.841 INFO:teuthology.orchestra.run.smithi090.stdout: ceph x86_64 2:17.2.7-684.g6716c9f3.el9 ceph 6.5 k 2024-03-18T01:08:08.841 INFO:teuthology.orchestra.run.smithi090.stdout:Installing dependencies: 2024-03-18T01:08:08.841 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mds x86_64 2:17.2.7-684.g6716c9f3.el9 ceph 2.1 M 2024-03-18T01:08:08.841 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mgr x86_64 2:17.2.7-684.g6716c9f3.el9 ceph 1.4 M 2024-03-18T01:08:08.841 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mgr-modules-core noarch 2:17.2.7-684.g6716c9f3.el9 ceph-noarch 239 k 2024-03-18T01:08:08.841 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mon x86_64 2:17.2.7-684.g6716c9f3.el9 ceph 3.7 M 2024-03-18T01:08:08.841 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-osd x86_64 2:17.2.7-684.g6716c9f3.el9 ceph 15 M 2024-03-18T01:08:08.841 INFO:teuthology.orchestra.run.smithi090.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-18T01:08:08.842 INFO:teuthology.orchestra.run.smithi090.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-18T01:08:08.842 INFO:teuthology.orchestra.run.smithi090.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-18T01:08:08.842 INFO:teuthology.orchestra.run.smithi090.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-18T01:08:08.842 INFO:teuthology.orchestra.run.smithi090.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-18T01:08:08.842 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-18T01:08:08.842 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-18T01:08:08.842 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-18T01:08:08.842 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-18T01:08:08.842 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-18T01:08:08.842 INFO:teuthology.orchestra.run.smithi090.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-18T01:08:08.842 INFO:teuthology.orchestra.run.smithi090.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-18T01:08:08.842 INFO:teuthology.orchestra.run.smithi090.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-18T01:08:08.843 INFO:teuthology.orchestra.run.smithi090.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-18T01:08:08.843 INFO:teuthology.orchestra.run.smithi090.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-18T01:08:08.843 INFO:teuthology.orchestra.run.smithi090.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-18T01:08:08.843 INFO:teuthology.orchestra.run.smithi090.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-18T01:08:08.843 INFO:teuthology.orchestra.run.smithi090.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-18T01:08:08.843 INFO:teuthology.orchestra.run.smithi090.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-18T01:08:08.843 INFO:teuthology.orchestra.run.smithi090.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-18T01:08:08.843 INFO:teuthology.orchestra.run.smithi090.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-18T01:08:08.843 INFO:teuthology.orchestra.run.smithi090.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-18T01:08:08.843 INFO:teuthology.orchestra.run.smithi090.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-18T01:08:08.843 INFO:teuthology.orchestra.run.smithi090.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-18T01:08:08.843 INFO:teuthology.orchestra.run.smithi090.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-18T01:08:08.843 INFO:teuthology.orchestra.run.smithi090.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-18T01:08:08.843 INFO:teuthology.orchestra.run.smithi090.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-18T01:08:08.843 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-18T01:08:08.844 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction Summary 2024-03-18T01:08:08.844 INFO:teuthology.orchestra.run.smithi090.stdout:======================================================================================== 2024-03-18T01:08:08.844 INFO:teuthology.orchestra.run.smithi090.stdout:Install 33 Packages 2024-03-18T01:08:08.844 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-18T01:08:08.844 INFO:teuthology.orchestra.run.smithi090.stdout:Total download size: 27 M 2024-03-18T01:08:08.844 INFO:teuthology.orchestra.run.smithi090.stdout:Installed size: 100 M 2024-03-18T01:08:08.845 INFO:teuthology.orchestra.run.smithi090.stdout:Downloading Packages: 2024-03-18T01:08:09.392 INFO:teuthology.orchestra.run.smithi090.stdout:(1/33): ceph-17.2.7-684.g6716c9f3.el9.x86_64.rp 33 kB/s | 6.5 kB 00:00 2024-03-18T01:08:09.592 INFO:teuthology.orchestra.run.smithi090.stdout:(2/33): ceph-mgr-17.2.7-684.g6716c9f3.el9.x86_6 3.5 MB/s | 1.4 MB 00:00 2024-03-18T01:08:09.826 INFO:teuthology.orchestra.run.smithi090.stdout:(3/33): ceph-mon-17.2.7-684.g6716c9f3.el9.x86_6 8.5 MB/s | 3.7 MB 00:00 2024-03-18T01:08:09.893 INFO:teuthology.orchestra.run.smithi090.stdout:(4/33): ceph-mgr-modules-core-17.2.7-684.g6716c 3.5 MB/s | 239 kB 00:00 2024-03-18T01:08:10.160 INFO:teuthology.orchestra.run.smithi090.stdout:(5/33): python3-mako-1.1.4-6.el9.noarch.rpm 541 kB/s | 144 kB 00:00 2024-03-18T01:08:10.227 INFO:teuthology.orchestra.run.smithi090.stdout:(6/33): python3-markupsafe-1.1.1-12.el9.x86_64. 475 kB/s | 32 kB 00:00 2024-03-18T01:08:10.427 INFO:teuthology.orchestra.run.smithi090.stdout:(7/33): ceph-osd-17.2.7-684.g6716c9f3.el9.x86_6 18 MB/s | 15 MB 00:00 2024-03-18T01:08:10.452 INFO:teuthology.orchestra.run.smithi090.stdout:(8/33): python3-pytz-2021.1-5.el9.noarch.rpm 210 kB/s | 47 kB 00:00 2024-03-18T01:08:10.494 INFO:teuthology.orchestra.run.smithi090.stdout:(9/33): python3-toml-0.10.2-6.el9.noarch.rpm 580 kB/s | 39 kB 00:00 2024-03-18T01:08:10.570 INFO:teuthology.orchestra.run.smithi090.stdout:(10/33): python3-cffi-1.14.5-5.el9.x86_64.rpm 2.0 MB/s | 241 kB 00:00 2024-03-18T01:08:10.687 INFO:teuthology.orchestra.run.smithi090.stdout:(11/33): python3-cryptography-36.0.1-4.el9.x86_ 6.0 MB/s | 1.1 MB 00:00 2024-03-18T01:08:10.712 INFO:teuthology.orchestra.run.smithi090.stdout:(12/33): python3-ply-3.11-14.el9.noarch.rpm 726 kB/s | 103 kB 00:00 2024-03-18T01:08:10.754 INFO:teuthology.orchestra.run.smithi090.stdout:(13/33): python3-pycparser-2.20-6.el9.noarch.rp 1.8 MB/s | 124 kB 00:00 2024-03-18T01:08:10.779 INFO:teuthology.orchestra.run.smithi090.stdout:(14/33): python3-urllib3-1.26.5-5.el9.noarch.rp 2.7 MB/s | 187 kB 00:00 2024-03-18T01:08:11.080 INFO:teuthology.orchestra.run.smithi090.stdout:(15/33): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 145 kB/s | 43 kB 00:00 2024-03-18T01:08:11.280 INFO:teuthology.orchestra.run.smithi090.stdout:(16/33): python3-cheroot-8.6.0-4.el9.noarch.rpm 859 kB/s | 172 kB 00:00 2024-03-18T01:08:11.307 INFO:teuthology.orchestra.run.smithi090.stdout:(17/33): python3-requests-2.25.1-8.el9.noarch.r 226 kB/s | 125 kB 00:00 2024-03-18T01:08:11.449 INFO:teuthology.orchestra.run.smithi090.stdout:(18/33): python3-cherrypy-18.6.1-2.el9.noarch.r 2.1 MB/s | 358 kB 00:00 2024-03-18T01:08:11.524 INFO:teuthology.orchestra.run.smithi090.stdout:(19/33): python3-jaraco-classes-3.2.1-5.el9.noa 238 kB/s | 18 kB 00:00 2024-03-18T01:08:11.549 INFO:teuthology.orchestra.run.smithi090.stdout:(20/33): python3-jaraco-8.2.1-3.el9.noarch.rpm 44 kB/s | 11 kB 00:00 2024-03-18T01:08:11.600 INFO:teuthology.orchestra.run.smithi090.stdout:(21/33): python3-jaraco-collections-3.0.0-8.el9 307 kB/s | 23 kB 00:00 2024-03-18T01:08:11.642 INFO:teuthology.orchestra.run.smithi090.stdout:(22/33): python3-jaraco-functools-3.5.0-2.el9.n 211 kB/s | 19 kB 00:00 2024-03-18T01:08:11.667 INFO:teuthology.orchestra.run.smithi090.stdout:(23/33): python3-jaraco-text-3.2.0-6.el9.noarch 294 kB/s | 20 kB 00:00 2024-03-18T01:08:11.751 INFO:teuthology.orchestra.run.smithi090.stdout:(24/33): python3-more-itertools-8.12.0-2.el9.no 948 kB/s | 79 kB 00:00 2024-03-18T01:08:11.784 INFO:teuthology.orchestra.run.smithi090.stdout:(25/33): python3-logutils-0.3.5-21.el9.noarch.r 325 kB/s | 46 kB 00:00 2024-03-18T01:08:11.851 INFO:teuthology.orchestra.run.smithi090.stdout:(26/33): python3-pecan-1.4.2-3.el9.noarch.rpm 2.7 MB/s | 272 kB 00:00 2024-03-18T01:08:11.877 INFO:teuthology.orchestra.run.smithi090.stdout:(27/33): python3-portend-3.1.0-2.el9.noarch.rpm 178 kB/s | 16 kB 00:00 2024-03-18T01:08:11.935 INFO:teuthology.orchestra.run.smithi090.stdout:(28/33): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.0 MB/s | 90 kB 00:00 2024-03-18T01:08:11.969 INFO:teuthology.orchestra.run.smithi090.stdout:(29/33): python3-tempora-5.0.0-2.el9.noarch.rpm 388 kB/s | 36 kB 00:00 2024-03-18T01:08:12.036 INFO:teuthology.orchestra.run.smithi090.stdout:(30/33): python3-webob-1.8.7-6.el9.noarch.rpm 2.2 MB/s | 230 kB 00:00 2024-03-18T01:08:12.081 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-selinux-2:17.2.7-684.g6716c9f3.el9.x86_64 23/26 2024-03-18T01:08:12.081 INFO:teuthology.orchestra.run.smithi029.stdout:skipping the directory /sys 2024-03-18T01:08:12.081 INFO:teuthology.orchestra.run.smithi029.stdout:skipping the directory /proc 2024-03-18T01:08:12.082 INFO:teuthology.orchestra.run.smithi029.stdout:skipping the directory /mnt 2024-03-18T01:08:12.082 INFO:teuthology.orchestra.run.smithi029.stdout:skipping the directory /var/tmp 2024-03-18T01:08:12.082 INFO:teuthology.orchestra.run.smithi029.stdout:skipping the directory /home 2024-03-18T01:08:12.082 INFO:teuthology.orchestra.run.smithi029.stdout:skipping the directory /root 2024-03-18T01:08:12.082 INFO:teuthology.orchestra.run.smithi029.stdout:skipping the directory /tmp 2024-03-18T01:08:12.082 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-18T01:08:12.086 INFO:teuthology.orchestra.run.smithi090.stdout:(31/33): ceph-mds-17.2.7-684.g6716c9f3.el9.x86_ 733 kB/s | 2.1 MB 00:02 2024-03-18T01:08:12.112 INFO:teuthology.orchestra.run.smithi090.stdout:(32/33): python3-zc-lockfile-2.0-10.el9.noarch. 265 kB/s | 20 kB 00:00 2024-03-18T01:08:12.237 INFO:teuthology.orchestra.run.smithi090.stdout:(33/33): python3-werkzeug-2.0.3-3.el9.1.noarch. 1.6 MB/s | 427 kB 00:00 2024-03-18T01:08:12.243 INFO:teuthology.orchestra.run.smithi090.stdout:-------------------------------------------------------------------------------- 2024-03-18T01:08:12.243 INFO:teuthology.orchestra.run.smithi090.stdout:Total 7.9 MB/s | 27 MB 00:03 2024-03-18T01:08:12.256 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction check 2024-03-18T01:08:12.297 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction check succeeded. 2024-03-18T01:08:12.297 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction test 2024-03-18T01:08:12.662 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction test succeeded. 2024-03-18T01:08:12.662 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction 2024-03-18T01:08:13.410 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-radosgw-2:17.2.7-684.g6716c9f3.el9.x86_64 24/26 2024-03-18T01:08:13.489 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-radosgw-2:17.2.7-684.g6716c9f3.el9.x86_64 24/26 2024-03-18T01:08:13.490 INFO:teuthology.orchestra.run.smithi029.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-18T01:08:13.490 INFO:teuthology.orchestra.run.smithi029.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-18T01:08:13.490 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-18T01:08:13.490 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-18T01:08:13.490 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-18T01:08:13.492 INFO:teuthology.orchestra.run.smithi029.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 25/26 2024-03-18T01:08:13.584 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 25/26 2024-03-18T01:08:13.584 INFO:teuthology.orchestra.run.smithi029.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 26/26 2024-03-18T01:08:13.931 INFO:teuthology.orchestra.run.smithi090.stdout: Preparing : 1/1 2024-03-18T01:08:13.999 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/33 2024-03-18T01:08:14.039 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/33 2024-03-18T01:08:14.089 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/33 2024-03-18T01:08:14.183 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/33 2024-03-18T01:08:14.262 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/33 2024-03-18T01:08:14.310 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 6/33 2024-03-18T01:08:14.383 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/33 2024-03-18T01:08:14.445 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 8/33 2024-03-18T01:08:14.491 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 9/33 2024-03-18T01:08:14.549 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-ply-3.11-14.el9.noarch 10/33 2024-03-18T01:08:14.622 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 11/33 2024-03-18T01:08:14.811 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 12/33 2024-03-18T01:08:14.865 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 13/33 2024-03-18T01:08:14.911 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 14/33 2024-03-18T01:08:14.946 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 15/33 2024-03-18T01:08:14.976 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 16/33 2024-03-18T01:08:15.053 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 17/33 2024-03-18T01:08:15.095 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 18/33 2024-03-18T01:08:15.130 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 19/33 2024-03-18T01:08:15.194 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 20/33 2024-03-18T01:08:15.211 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 26/26 2024-03-18T01:08:15.211 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-base-2:17.2.7-684.g6716c9f3.el9.x86_64 1/26 2024-03-18T01:08:15.211 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-common-2:17.2.7-684.g6716c9f3.el9.x86_64 2/26 2024-03-18T01:08:15.211 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-radosgw-2:17.2.7-684.g6716c9f3.el9.x86_64 3/26 2024-03-18T01:08:15.211 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-selinux-2:17.2.7-684.g6716c9f3.el9.x86_64 4/26 2024-03-18T01:08:15.211 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libcephfs2-2:17.2.7-684.g6716c9f3.el9.x86_64 5/26 2024-03-18T01:08:15.211 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libradosstriper1-2:17.2.7-684.g6716c9f3.el9.x86_64 6/26 2024-03-18T01:08:15.211 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : librgw2-2:17.2.7-684.g6716c9f3.el9.x86_64 7/26 2024-03-18T01:08:15.212 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-ceph-argparse-2:17.2.7-684.g6716c9f3.el9.x 8/26 2024-03-18T01:08:15.212 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-ceph-common-2:17.2.7-684.g6716c9f3.el9.x86 9/26 2024-03-18T01:08:15.212 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-cephfs-2:17.2.7-684.g6716c9f3.el9.x86_64 10/26 2024-03-18T01:08:15.212 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-rados-2:17.2.7-684.g6716c9f3.el9.x86_64 11/26 2024-03-18T01:08:15.212 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-rbd-2:17.2.7-684.g6716c9f3.el9.x86_64 12/26 2024-03-18T01:08:15.212 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-rgw-2:17.2.7-684.g6716c9f3.el9.x86_64 13/26 2024-03-18T01:08:15.212 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 14/26 2024-03-18T01:08:15.212 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 15/26 2024-03-18T01:08:15.212 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 16/26 2024-03-18T01:08:15.213 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 17/26 2024-03-18T01:08:15.213 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 18/26 2024-03-18T01:08:15.213 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 19/26 2024-03-18T01:08:15.213 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 20/26 2024-03-18T01:08:15.214 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 21/26 2024-03-18T01:08:15.214 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 22/26 2024-03-18T01:08:15.214 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : librados2-2:17.2.7-684.g6716c9f3.el9.x86_64 23/26 2024-03-18T01:08:15.214 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 24/26 2024-03-18T01:08:15.214 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : librbd1-2:17.2.7-684.g6716c9f3.el9.x86_64 25/26 2024-03-18T01:08:15.238 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 21/33 2024-03-18T01:08:15.274 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 22/33 2024-03-18T01:08:15.365 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 23/33 2024-03-18T01:08:15.405 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 24/33 2024-03-18T01:08:15.478 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 25/33 2024-03-18T01:08:15.623 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 26/33 2024-03-18T01:08:15.748 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 27/33 2024-03-18T01:08:15.898 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-mgr-modules-core-2:17.2.7-684.g6716c9f3.el9.n 28/33 2024-03-18T01:08:15.932 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-mgr-2:17.2.7-684.g6716c9f3.el9.x86_64 29/33 2024-03-18T01:08:15.953 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-mgr-2:17.2.7-684.g6716c9f3.el9.x86_64 29/33 2024-03-18T01:08:15.954 INFO:teuthology.orchestra.run.smithi090.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-18T01:08:15.954 INFO:teuthology.orchestra.run.smithi090.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-18T01:08:15.954 INFO:teuthology.orchestra.run.smithi090.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-18T01:08:15.954 INFO:teuthology.orchestra.run.smithi090.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-18T01:08:15.954 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-18T01:08:16.167 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 26/26 2024-03-18T01:08:16.167 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-18T01:08:16.167 INFO:teuthology.orchestra.run.smithi029.stdout:Upgraded: 2024-03-18T01:08:16.167 INFO:teuthology.orchestra.run.smithi029.stdout: librados2-2:17.2.7-684.g6716c9f3.el9.x86_64 2024-03-18T01:08:16.167 INFO:teuthology.orchestra.run.smithi029.stdout: librbd1-2:17.2.7-684.g6716c9f3.el9.x86_64 2024-03-18T01:08:16.167 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2024-03-18T01:08:16.167 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-base-2:17.2.7-684.g6716c9f3.el9.x86_64 2024-03-18T01:08:16.167 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-common-2:17.2.7-684.g6716c9f3.el9.x86_64 2024-03-18T01:08:16.167 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-radosgw-2:17.2.7-684.g6716c9f3.el9.x86_64 2024-03-18T01:08:16.167 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-selinux-2:17.2.7-684.g6716c9f3.el9.x86_64 2024-03-18T01:08:16.167 INFO:teuthology.orchestra.run.smithi029.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-18T01:08:16.167 INFO:teuthology.orchestra.run.smithi029.stdout: libcephfs2-2:17.2.7-684.g6716c9f3.el9.x86_64 2024-03-18T01:08:16.168 INFO:teuthology.orchestra.run.smithi029.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-18T01:08:16.168 INFO:teuthology.orchestra.run.smithi029.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-18T01:08:16.168 INFO:teuthology.orchestra.run.smithi029.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-18T01:08:16.168 INFO:teuthology.orchestra.run.smithi029.stdout: libradosstriper1-2:17.2.7-684.g6716c9f3.el9.x86_64 2024-03-18T01:08:16.168 INFO:teuthology.orchestra.run.smithi029.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-18T01:08:16.168 INFO:teuthology.orchestra.run.smithi029.stdout: librgw2-2:17.2.7-684.g6716c9f3.el9.x86_64 2024-03-18T01:08:16.168 INFO:teuthology.orchestra.run.smithi029.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-18T01:08:16.168 INFO:teuthology.orchestra.run.smithi029.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-18T01:08:16.168 INFO:teuthology.orchestra.run.smithi029.stdout: python3-ceph-argparse-2:17.2.7-684.g6716c9f3.el9.x86_64 2024-03-18T01:08:16.168 INFO:teuthology.orchestra.run.smithi029.stdout: python3-ceph-common-2:17.2.7-684.g6716c9f3.el9.x86_64 2024-03-18T01:08:16.168 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cephfs-2:17.2.7-684.g6716c9f3.el9.x86_64 2024-03-18T01:08:16.168 INFO:teuthology.orchestra.run.smithi029.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-18T01:08:16.168 INFO:teuthology.orchestra.run.smithi029.stdout: python3-rados-2:17.2.7-684.g6716c9f3.el9.x86_64 2024-03-18T01:08:16.168 INFO:teuthology.orchestra.run.smithi029.stdout: python3-rbd-2:17.2.7-684.g6716c9f3.el9.x86_64 2024-03-18T01:08:16.168 INFO:teuthology.orchestra.run.smithi029.stdout: python3-rgw-2:17.2.7-684.g6716c9f3.el9.x86_64 2024-03-18T01:08:16.169 INFO:teuthology.orchestra.run.smithi029.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-18T01:08:16.169 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-18T01:08:16.169 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2024-03-18T01:08:16.576 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install ceph-test 2024-03-18T01:08:17.147 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:00:31 ago on Mon 18 Mar 2024 01:07:46 AM UTC. 2024-03-18T01:08:17.309 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2024-03-18T01:08:17.309 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-03-18T01:08:17.309 INFO:teuthology.orchestra.run.smithi029.stdout: Package Arch Version Repository Size 2024-03-18T01:08:17.309 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-03-18T01:08:17.310 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2024-03-18T01:08:17.310 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-test x86_64 2:17.2.7-684.g6716c9f3.el9 ceph 48 M 2024-03-18T01:08:17.310 INFO:teuthology.orchestra.run.smithi029.stdout:Installing dependencies: 2024-03-18T01:08:17.310 INFO:teuthology.orchestra.run.smithi029.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-18T01:08:17.310 INFO:teuthology.orchestra.run.smithi029.stdout: libcephsqlite x86_64 2:17.2.7-684.g6716c9f3.el9 ceph 163 k 2024-03-18T01:08:17.310 INFO:teuthology.orchestra.run.smithi029.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-18T01:08:17.310 INFO:teuthology.orchestra.run.smithi029.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-18T01:08:17.310 INFO:teuthology.orchestra.run.smithi029.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-18T01:08:17.310 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-18T01:08:17.310 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2024-03-18T01:08:17.310 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-03-18T01:08:17.310 INFO:teuthology.orchestra.run.smithi029.stdout:Install 6 Packages 2024-03-18T01:08:17.310 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-18T01:08:17.311 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 49 M 2024-03-18T01:08:17.311 INFO:teuthology.orchestra.run.smithi029.stdout:Installed size: 196 M 2024-03-18T01:08:17.311 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2024-03-18T01:08:17.363 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-osd-2:17.2.7-684.g6716c9f3.el9.x86_64 30/33 2024-03-18T01:08:17.390 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-osd-2:17.2.7-684.g6716c9f3.el9.x86_64 30/33 2024-03-18T01:08:17.390 INFO:teuthology.orchestra.run.smithi090.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-18T01:08:17.390 INFO:teuthology.orchestra.run.smithi090.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-18T01:08:17.390 INFO:teuthology.orchestra.run.smithi090.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-18T01:08:17.390 INFO:teuthology.orchestra.run.smithi090.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-18T01:08:17.391 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-18T01:08:17.773 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-mon-2:17.2.7-684.g6716c9f3.el9.x86_64 31/33 2024-03-18T01:08:17.795 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-mon-2:17.2.7-684.g6716c9f3.el9.x86_64 31/33 2024-03-18T01:08:17.795 INFO:teuthology.orchestra.run.smithi090.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-18T01:08:17.795 INFO:teuthology.orchestra.run.smithi090.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-18T01:08:17.795 INFO:teuthology.orchestra.run.smithi090.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-18T01:08:17.795 INFO:teuthology.orchestra.run.smithi090.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-18T01:08:17.795 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-18T01:08:17.875 INFO:teuthology.orchestra.run.smithi029.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 887 kB/s | 217 kB 00:00 2024-03-18T01:08:17.959 INFO:teuthology.orchestra.run.smithi029.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 3.5 MB/s | 300 kB 00:00 2024-03-18T01:08:17.998 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-mds-2:17.2.7-684.g6716c9f3.el9.x86_64 32/33 2024-03-18T01:08:18.009 INFO:teuthology.orchestra.run.smithi029.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.2 MB/s | 63 kB 00:00 2024-03-18T01:08:18.021 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-mds-2:17.2.7-684.g6716c9f3.el9.x86_64 32/33 2024-03-18T01:08:18.022 INFO:teuthology.orchestra.run.smithi090.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-18T01:08:18.022 INFO:teuthology.orchestra.run.smithi090.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-18T01:08:18.022 INFO:teuthology.orchestra.run.smithi090.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-18T01:08:18.022 INFO:teuthology.orchestra.run.smithi090.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-18T01:08:18.022 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-18T01:08:18.088 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-2:17.2.7-684.g6716c9f3.el9.x86_64 33/33 2024-03-18T01:08:18.160 INFO:teuthology.orchestra.run.smithi029.stdout:(4/6): libcephsqlite-17.2.7-684.g6716c9f3.el9.x 307 kB/s | 163 kB 00:00 2024-03-18T01:08:18.185 INFO:teuthology.orchestra.run.smithi029.stdout:(5/6): jq-1.6-16.el9.x86_64.rpm 1.0 MB/s | 187 kB 00:00 2024-03-18T01:08:20.176 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-2:17.2.7-684.g6716c9f3.el9.x86_64 33/33 2024-03-18T01:08:20.176 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-2:17.2.7-684.g6716c9f3.el9.x86_64 1/33 2024-03-18T01:08:20.177 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-mds-2:17.2.7-684.g6716c9f3.el9.x86_64 2/33 2024-03-18T01:08:20.177 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-mgr-2:17.2.7-684.g6716c9f3.el9.x86_64 3/33 2024-03-18T01:08:20.177 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-mon-2:17.2.7-684.g6716c9f3.el9.x86_64 4/33 2024-03-18T01:08:20.177 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-osd-2:17.2.7-684.g6716c9f3.el9.x86_64 5/33 2024-03-18T01:08:20.177 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-mgr-modules-core-2:17.2.7-684.g6716c9f3.el9.n 6/33 2024-03-18T01:08:20.177 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 7/33 2024-03-18T01:08:20.177 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 8/33 2024-03-18T01:08:20.177 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 9/33 2024-03-18T01:08:20.177 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 10/33 2024-03-18T01:08:20.177 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 11/33 2024-03-18T01:08:20.177 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 12/33 2024-03-18T01:08:20.177 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-ply-3.11-14.el9.noarch 13/33 2024-03-18T01:08:20.177 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 14/33 2024-03-18T01:08:20.177 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 15/33 2024-03-18T01:08:20.178 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 16/33 2024-03-18T01:08:20.178 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 17/33 2024-03-18T01:08:20.178 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 18/33 2024-03-18T01:08:20.178 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 19/33 2024-03-18T01:08:20.178 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 20/33 2024-03-18T01:08:20.178 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 21/33 2024-03-18T01:08:20.178 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 22/33 2024-03-18T01:08:20.178 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 23/33 2024-03-18T01:08:20.178 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 24/33 2024-03-18T01:08:20.178 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 25/33 2024-03-18T01:08:20.178 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 26/33 2024-03-18T01:08:20.178 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 27/33 2024-03-18T01:08:20.179 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 28/33 2024-03-18T01:08:20.179 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 29/33 2024-03-18T01:08:20.179 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 30/33 2024-03-18T01:08:20.179 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 31/33 2024-03-18T01:08:20.179 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 32/33 2024-03-18T01:08:21.202 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 33/33 2024-03-18T01:08:21.321 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-18T01:08:21.321 INFO:teuthology.orchestra.run.smithi090.stdout:Installed: 2024-03-18T01:08:21.321 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-2:17.2.7-684.g6716c9f3.el9.x86_64 2024-03-18T01:08:21.321 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mds-2:17.2.7-684.g6716c9f3.el9.x86_64 2024-03-18T01:08:21.321 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mgr-2:17.2.7-684.g6716c9f3.el9.x86_64 2024-03-18T01:08:21.321 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mgr-modules-core-2:17.2.7-684.g6716c9f3.el9.noarch 2024-03-18T01:08:21.321 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mon-2:17.2.7-684.g6716c9f3.el9.x86_64 2024-03-18T01:08:21.321 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-osd-2:17.2.7-684.g6716c9f3.el9.x86_64 2024-03-18T01:08:21.321 INFO:teuthology.orchestra.run.smithi090.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-18T01:08:21.322 INFO:teuthology.orchestra.run.smithi090.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-18T01:08:21.322 INFO:teuthology.orchestra.run.smithi090.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-18T01:08:21.322 INFO:teuthology.orchestra.run.smithi090.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-18T01:08:21.322 INFO:teuthology.orchestra.run.smithi090.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-18T01:08:21.322 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-18T01:08:21.322 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-18T01:08:21.322 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-18T01:08:21.322 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-18T01:08:21.322 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-18T01:08:21.322 INFO:teuthology.orchestra.run.smithi090.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-18T01:08:21.322 INFO:teuthology.orchestra.run.smithi090.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-18T01:08:21.322 INFO:teuthology.orchestra.run.smithi090.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-18T01:08:21.322 INFO:teuthology.orchestra.run.smithi090.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-18T01:08:21.322 INFO:teuthology.orchestra.run.smithi090.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-18T01:08:21.323 INFO:teuthology.orchestra.run.smithi090.stdout: python3-ply-3.11-14.el9.noarch 2024-03-18T01:08:21.323 INFO:teuthology.orchestra.run.smithi090.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-18T01:08:21.323 INFO:teuthology.orchestra.run.smithi090.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-18T01:08:21.323 INFO:teuthology.orchestra.run.smithi090.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-18T01:08:21.323 INFO:teuthology.orchestra.run.smithi090.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-18T01:08:21.323 INFO:teuthology.orchestra.run.smithi090.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-18T01:08:21.323 INFO:teuthology.orchestra.run.smithi090.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-18T01:08:21.323 INFO:teuthology.orchestra.run.smithi090.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-18T01:08:21.323 INFO:teuthology.orchestra.run.smithi090.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-18T01:08:21.323 INFO:teuthology.orchestra.run.smithi090.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-18T01:08:21.323 INFO:teuthology.orchestra.run.smithi090.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-18T01:08:21.323 INFO:teuthology.orchestra.run.smithi090.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-18T01:08:21.323 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-18T01:08:21.323 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-03-18T01:08:21.642 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install ceph-base 2024-03-18T01:08:22.214 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:00:55 ago on Mon 18 Mar 2024 01:07:27 AM UTC. 2024-03-18T01:08:22.311 INFO:teuthology.orchestra.run.smithi090.stdout:Package ceph-base-2:17.2.7-684.g6716c9f3.el9.x86_64 is already installed. 2024-03-18T01:08:22.363 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-03-18T01:08:22.364 INFO:teuthology.orchestra.run.smithi090.stdout:Nothing to do. 2024-03-18T01:08:22.364 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-03-18T01:08:22.431 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install cephadm 2024-03-18T01:08:22.985 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:00:55 ago on Mon 18 Mar 2024 01:07:27 AM UTC. 2024-03-18T01:08:23.133 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-03-18T01:08:23.134 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-03-18T01:08:23.134 INFO:teuthology.orchestra.run.smithi090.stdout: Package Arch Version Repository Size 2024-03-18T01:08:23.134 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-03-18T01:08:23.134 INFO:teuthology.orchestra.run.smithi090.stdout:Installing: 2024-03-18T01:08:23.134 INFO:teuthology.orchestra.run.smithi090.stdout: cephadm noarch 2:17.2.7-684.g6716c9f3.el9 ceph-noarch 82 k 2024-03-18T01:08:23.134 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-18T01:08:23.134 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction Summary 2024-03-18T01:08:23.135 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-03-18T01:08:23.135 INFO:teuthology.orchestra.run.smithi090.stdout:Install 1 Package 2024-03-18T01:08:23.135 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-18T01:08:23.135 INFO:teuthology.orchestra.run.smithi090.stdout:Total download size: 82 k 2024-03-18T01:08:23.135 INFO:teuthology.orchestra.run.smithi090.stdout:Installed size: 372 k 2024-03-18T01:08:23.135 INFO:teuthology.orchestra.run.smithi090.stdout:Downloading Packages: 2024-03-18T01:08:23.528 INFO:teuthology.orchestra.run.smithi090.stdout:cephadm-17.2.7-684.g6716c9f3.el9.noarch.rpm 209 kB/s | 82 kB 00:00 2024-03-18T01:08:23.528 INFO:teuthology.orchestra.run.smithi090.stdout:-------------------------------------------------------------------------------- 2024-03-18T01:08:23.529 INFO:teuthology.orchestra.run.smithi090.stdout:Total 208 kB/s | 82 kB 00:00 2024-03-18T01:08:23.529 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction check 2024-03-18T01:08:23.534 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction check succeeded. 2024-03-18T01:08:23.534 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction test 2024-03-18T01:08:23.536 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction test succeeded. 2024-03-18T01:08:23.536 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction 2024-03-18T01:08:23.612 INFO:teuthology.orchestra.run.smithi090.stdout: Preparing : 1/1 2024-03-18T01:08:23.686 INFO:teuthology.orchestra.run.smithi029.stdout:(6/6): ceph-test-17.2.7-684.g6716c9f3.el9.x86_6 7.9 MB/s | 48 MB 00:06 2024-03-18T01:08:23.688 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2024-03-18T01:08:23.688 INFO:teuthology.orchestra.run.smithi029.stdout:Total 7.7 MB/s | 49 MB 00:06 2024-03-18T01:08:23.701 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2024-03-18T01:08:23.720 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2024-03-18T01:08:23.720 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2024-03-18T01:08:24.077 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: cephadm-2:17.2.7-684.g6716c9f3.el9.noarch 1/1 2024-03-18T01:08:24.124 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2024-03-18T01:08:24.124 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2024-03-18T01:08:24.160 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : cephadm-2:17.2.7-684.g6716c9f3.el9.noarch 1/1 2024-03-18T01:08:24.634 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: cephadm-2:17.2.7-684.g6716c9f3.el9.noarch 1/1 2024-03-18T01:08:24.725 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2024-03-18T01:08:24.805 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-18T01:08:24.852 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-18T01:08:24.896 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-18T01:08:24.931 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : cephadm-2:17.2.7-684.g6716c9f3.el9.noarch 1/1 2024-03-18T01:08:24.931 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-18T01:08:24.931 INFO:teuthology.orchestra.run.smithi090.stdout:Installed: 2024-03-18T01:08:24.931 INFO:teuthology.orchestra.run.smithi090.stdout: cephadm-2:17.2.7-684.g6716c9f3.el9.noarch 2024-03-18T01:08:24.931 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-18T01:08:24.931 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-03-18T01:08:24.949 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-18T01:08:24.985 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : libcephsqlite-2:17.2.7-684.g6716c9f3.el9.x86_64 5/6 2024-03-18T01:08:25.192 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install ceph-immutable-object-cache 2024-03-18T01:08:25.762 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:00:58 ago on Mon 18 Mar 2024 01:07:27 AM UTC. 2024-03-18T01:08:25.915 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-03-18T01:08:25.915 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-03-18T01:08:25.916 INFO:teuthology.orchestra.run.smithi090.stdout: Package Arch Version Repo Size 2024-03-18T01:08:25.916 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-03-18T01:08:25.916 INFO:teuthology.orchestra.run.smithi090.stdout:Installing: 2024-03-18T01:08:25.916 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-immutable-object-cache x86_64 2:17.2.7-684.g6716c9f3.el9 ceph 137 k 2024-03-18T01:08:25.916 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-18T01:08:25.916 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction Summary 2024-03-18T01:08:25.916 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-03-18T01:08:25.916 INFO:teuthology.orchestra.run.smithi090.stdout:Install 1 Package 2024-03-18T01:08:25.916 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-18T01:08:25.917 INFO:teuthology.orchestra.run.smithi090.stdout:Total download size: 137 k 2024-03-18T01:08:25.917 INFO:teuthology.orchestra.run.smithi090.stdout:Installed size: 379 k 2024-03-18T01:08:25.917 INFO:teuthology.orchestra.run.smithi090.stdout:Downloading Packages: 2024-03-18T01:08:26.230 INFO:teuthology.orchestra.run.smithi090.stdout:ceph-immutable-object-cache-17.2.7-684.g6716c9f 438 kB/s | 137 kB 00:00 2024-03-18T01:08:26.231 INFO:teuthology.orchestra.run.smithi090.stdout:-------------------------------------------------------------------------------- 2024-03-18T01:08:26.231 INFO:teuthology.orchestra.run.smithi090.stdout:Total 436 kB/s | 137 kB 00:00 2024-03-18T01:08:26.231 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction check 2024-03-18T01:08:26.239 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction check succeeded. 2024-03-18T01:08:26.239 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction test 2024-03-18T01:08:26.291 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction test succeeded. 2024-03-18T01:08:26.291 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction 2024-03-18T01:08:26.428 INFO:teuthology.orchestra.run.smithi090.stdout: Preparing : 1/1 2024-03-18T01:08:26.490 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-immutable-object-cache-2:17.2.7-684.g6716c9f3.e 1/1 2024-03-18T01:08:26.517 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.2.7-684.g6716c9f3.e 1/1 2024-03-18T01:08:26.517 INFO:teuthology.orchestra.run.smithi090.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-18T01:08:26.517 INFO:teuthology.orchestra.run.smithi090.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-18T01:08:26.517 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-18T01:08:27.579 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-immutable-object-cache-2:17.2.7-684.g6716c9f3.e 1/1 2024-03-18T01:08:27.579 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-18T01:08:27.579 INFO:teuthology.orchestra.run.smithi090.stdout:Installed: 2024-03-18T01:08:27.579 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-immutable-object-cache-2:17.2.7-684.g6716c9f3.el9.x86_64 2024-03-18T01:08:27.579 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-18T01:08:27.579 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-03-18T01:08:27.860 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install ceph-mgr 2024-03-18T01:08:28.436 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:01:01 ago on Mon 18 Mar 2024 01:07:27 AM UTC. 2024-03-18T01:08:28.533 INFO:teuthology.orchestra.run.smithi090.stdout:Package ceph-mgr-2:17.2.7-684.g6716c9f3.el9.x86_64 is already installed. 2024-03-18T01:08:28.586 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-03-18T01:08:28.587 INFO:teuthology.orchestra.run.smithi090.stdout:Nothing to do. 2024-03-18T01:08:28.587 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-03-18T01:08:28.649 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install ceph-mgr-dashboard 2024-03-18T01:08:29.200 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:01:02 ago on Mon 18 Mar 2024 01:07:27 AM UTC. 2024-03-18T01:08:29.348 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-03-18T01:08:29.349 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-03-18T01:08:29.349 INFO:teuthology.orchestra.run.smithi090.stdout: Package Arch Version Repository Size 2024-03-18T01:08:29.349 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-03-18T01:08:29.349 INFO:teuthology.orchestra.run.smithi090.stdout:Installing: 2024-03-18T01:08:29.349 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mgr-dashboard noarch 2:17.2.7-684.g6716c9f3.el9 ceph-noarch 1.7 M 2024-03-18T01:08:29.349 INFO:teuthology.orchestra.run.smithi090.stdout:Installing dependencies: 2024-03-18T01:08:29.349 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-grafana-dashboards noarch 2:17.2.7-684.g6716c9f3.el9 ceph-noarch 23 k 2024-03-18T01:08:29.349 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-prometheus-alerts noarch 2:17.2.7-684.g6716c9f3.el9 ceph-noarch 15 k 2024-03-18T01:08:29.349 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2024-03-18T01:08:29.350 INFO:teuthology.orchestra.run.smithi090.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-18T01:08:29.350 INFO:teuthology.orchestra.run.smithi090.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-18T01:08:29.350 INFO:teuthology.orchestra.run.smithi090.stdout:Installing weak dependencies: 2024-03-18T01:08:29.350 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2024-03-18T01:08:29.350 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-18T01:08:29.350 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction Summary 2024-03-18T01:08:29.350 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-03-18T01:08:29.350 INFO:teuthology.orchestra.run.smithi090.stdout:Install 7 Packages 2024-03-18T01:08:29.350 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-18T01:08:29.351 INFO:teuthology.orchestra.run.smithi090.stdout:Total download size: 2.0 M 2024-03-18T01:08:29.351 INFO:teuthology.orchestra.run.smithi090.stdout:Installed size: 8.1 M 2024-03-18T01:08:29.351 INFO:teuthology.orchestra.run.smithi090.stdout:Downloading Packages: 2024-03-18T01:08:29.645 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: libcephsqlite-2:17.2.7-684.g6716c9f3.el9.x86_64 5/6 2024-03-18T01:08:29.714 INFO:teuthology.orchestra.run.smithi090.stdout:(1/7): ceph-prometheus-alerts-17.2.7-684.g6716c 64 kB/s | 15 kB 00:00 2024-03-18T01:08:29.733 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-test-2:17.2.7-684.g6716c9f3.el9.x86_64 6/6 2024-03-18T01:08:29.848 INFO:teuthology.orchestra.run.smithi090.stdout:(2/7): ceph-grafana-dashboards-17.2.7-684.g6716 65 kB/s | 23 kB 00:00 2024-03-18T01:08:29.873 INFO:teuthology.orchestra.run.smithi090.stdout:(3/7): python3-jwt+crypto-2.4.0-1.el9.noarch.rp 57 kB/s | 9.0 kB 00:00 2024-03-18T01:08:29.982 INFO:teuthology.orchestra.run.smithi090.stdout:(4/7): python3-jwt-2.4.0-1.el9.noarch.rpm 306 kB/s | 41 kB 00:00 2024-03-18T01:08:30.057 INFO:teuthology.orchestra.run.smithi090.stdout:(5/7): python3-repoze-lru-0.7-16.el9.noarch.rpm 167 kB/s | 31 kB 00:00 2024-03-18T01:08:30.183 INFO:teuthology.orchestra.run.smithi090.stdout:(6/7): python3-routes-2.5.1-5.el9.noarch.rpm 940 kB/s | 188 kB 00:00 2024-03-18T01:08:30.416 INFO:teuthology.orchestra.run.smithi090.stdout:(7/7): ceph-mgr-dashboard-17.2.7-684.g6716c9f3. 1.8 MB/s | 1.7 MB 00:00 2024-03-18T01:08:30.418 INFO:teuthology.orchestra.run.smithi090.stdout:-------------------------------------------------------------------------------- 2024-03-18T01:08:30.418 INFO:teuthology.orchestra.run.smithi090.stdout:Total 1.9 MB/s | 2.0 MB 00:01 2024-03-18T01:08:30.418 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction check 2024-03-18T01:08:30.426 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction check succeeded. 2024-03-18T01:08:30.426 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction test 2024-03-18T01:08:30.498 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction test succeeded. 2024-03-18T01:08:30.498 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction 2024-03-18T01:08:30.794 INFO:teuthology.orchestra.run.smithi090.stdout: Preparing : 1/1 2024-03-18T01:08:30.868 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 1/7 2024-03-18T01:08:30.901 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 2/7 2024-03-18T01:08:30.949 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/7 2024-03-18T01:08:30.997 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/7 2024-03-18T01:08:31.026 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-prometheus-alerts-2:17.2.7-684.g6716c9f3.el9.no 5/7 2024-03-18T01:08:31.256 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-grafana-dashboards-2:17.2.7-684.g6716c9f3.el9.n 6/7 2024-03-18T01:08:31.287 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-mgr-dashboard-2:17.2.7-684.g6716c9f3.el9.noarch 7/7 2024-03-18T01:08:31.928 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.2.7-684.g6716c9f3.el9.noarch 7/7 2024-03-18T01:08:31.929 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-grafana-dashboards-2:17.2.7-684.g6716c9f3.el9.n 1/7 2024-03-18T01:08:31.929 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-mgr-dashboard-2:17.2.7-684.g6716c9f3.el9.noarch 2/7 2024-03-18T01:08:31.929 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-prometheus-alerts-2:17.2.7-684.g6716c9f3.el9.no 3/7 2024-03-18T01:08:31.929 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 4/7 2024-03-18T01:08:31.929 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 5/7 2024-03-18T01:08:31.929 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 6/7 2024-03-18T01:08:32.234 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 7/7 2024-03-18T01:08:32.234 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-18T01:08:32.234 INFO:teuthology.orchestra.run.smithi090.stdout:Installed: 2024-03-18T01:08:32.234 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-grafana-dashboards-2:17.2.7-684.g6716c9f3.el9.noarch 2024-03-18T01:08:32.234 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mgr-dashboard-2:17.2.7-684.g6716c9f3.el9.noarch 2024-03-18T01:08:32.235 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-prometheus-alerts-2:17.2.7-684.g6716c9f3.el9.noarch 2024-03-18T01:08:32.235 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jwt-2.4.0-1.el9.noarch 2024-03-18T01:08:32.235 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2024-03-18T01:08:32.235 INFO:teuthology.orchestra.run.smithi090.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-18T01:08:32.235 INFO:teuthology.orchestra.run.smithi090.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-18T01:08:32.235 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-18T01:08:32.235 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-03-18T01:08:32.508 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-18T01:08:33.062 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:01:06 ago on Mon 18 Mar 2024 01:07:27 AM UTC. 2024-03-18T01:08:33.211 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-03-18T01:08:33.212 INFO:teuthology.orchestra.run.smithi090.stdout:=========================================================================================== 2024-03-18T01:08:33.212 INFO:teuthology.orchestra.run.smithi090.stdout: Package Arch Version Repository Size 2024-03-18T01:08:33.212 INFO:teuthology.orchestra.run.smithi090.stdout:=========================================================================================== 2024-03-18T01:08:33.213 INFO:teuthology.orchestra.run.smithi090.stdout:Installing: 2024-03-18T01:08:33.213 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mgr-diskprediction-local noarch 2:17.2.7-684.g6716c9f3.el9 ceph-noarch 7.4 M 2024-03-18T01:08:33.213 INFO:teuthology.orchestra.run.smithi090.stdout:Installing dependencies: 2024-03-18T01:08:33.213 INFO:teuthology.orchestra.run.smithi090.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-18T01:08:33.213 INFO:teuthology.orchestra.run.smithi090.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-18T01:08:33.213 INFO:teuthology.orchestra.run.smithi090.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-18T01:08:33.213 INFO:teuthology.orchestra.run.smithi090.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-18T01:08:33.213 INFO:teuthology.orchestra.run.smithi090.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-18T01:08:33.213 INFO:teuthology.orchestra.run.smithi090.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-18T01:08:33.213 INFO:teuthology.orchestra.run.smithi090.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-18T01:08:33.213 INFO:teuthology.orchestra.run.smithi090.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-18T01:08:33.213 INFO:teuthology.orchestra.run.smithi090.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-18T01:08:33.213 INFO:teuthology.orchestra.run.smithi090.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-18T01:08:33.214 INFO:teuthology.orchestra.run.smithi090.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-18T01:08:33.214 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-18T01:08:33.214 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction Summary 2024-03-18T01:08:33.214 INFO:teuthology.orchestra.run.smithi090.stdout:=========================================================================================== 2024-03-18T01:08:33.214 INFO:teuthology.orchestra.run.smithi090.stdout:Install 12 Packages 2024-03-18T01:08:33.214 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-18T01:08:33.214 INFO:teuthology.orchestra.run.smithi090.stdout:Total download size: 37 M 2024-03-18T01:08:33.215 INFO:teuthology.orchestra.run.smithi090.stdout:Installed size: 209 M 2024-03-18T01:08:33.215 INFO:teuthology.orchestra.run.smithi090.stdout:Downloading Packages: 2024-03-18T01:08:33.566 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-test-2:17.2.7-684.g6716c9f3.el9.x86_64 6/6 2024-03-18T01:08:33.566 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-test-2:17.2.7-684.g6716c9f3.el9.x86_64 1/6 2024-03-18T01:08:33.566 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libcephsqlite-2:17.2.7-684.g6716c9f3.el9.x86_64 2/6 2024-03-18T01:08:33.566 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-18T01:08:33.566 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-18T01:08:33.566 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-18T01:08:33.710 INFO:teuthology.orchestra.run.smithi090.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 228 kB/s | 32 kB 00:00 2024-03-18T01:08:33.743 INFO:teuthology.orchestra.run.smithi090.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 533 kB/s | 17 kB 00:00 2024-03-18T01:08:33.793 INFO:teuthology.orchestra.run.smithi090.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 701 kB/s | 34 kB 00:00 2024-03-18T01:08:33.852 INFO:teuthology.orchestra.run.smithi090.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 11 MB/s | 3.0 MB 00:00 2024-03-18T01:08:33.951 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-18T01:08:33.951 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-18T01:08:33.951 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2024-03-18T01:08:33.951 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-test-2:17.2.7-684.g6716c9f3.el9.x86_64 2024-03-18T01:08:33.951 INFO:teuthology.orchestra.run.smithi029.stdout: jq-1.6-16.el9.x86_64 2024-03-18T01:08:33.951 INFO:teuthology.orchestra.run.smithi029.stdout: libcephsqlite-2:17.2.7-684.g6716c9f3.el9.x86_64 2024-03-18T01:08:33.952 INFO:teuthology.orchestra.run.smithi029.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-18T01:08:33.952 INFO:teuthology.orchestra.run.smithi029.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-18T01:08:33.952 INFO:teuthology.orchestra.run.smithi029.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-18T01:08:33.952 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-18T01:08:33.952 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2024-03-18T01:08:34.096 INFO:teuthology.orchestra.run.smithi090.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 16 MB/s | 4.7 MB 00:00 2024-03-18T01:08:34.169 INFO:teuthology.orchestra.run.smithi090.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 15 MB/s | 4.8 MB 00:00 2024-03-18T01:08:34.194 INFO:teuthology.orchestra.run.smithi090.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.2 MB/s | 217 kB 00:00 2024-03-18T01:08:34.319 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install ceph 2024-03-18T01:08:34.520 INFO:teuthology.orchestra.run.smithi090.stdout:(8/12): ceph-mgr-diskprediction-local-17.2.7-68 7.8 MB/s | 7.4 MB 00:00 2024-03-18T01:08:34.704 INFO:teuthology.orchestra.run.smithi090.stdout:(9/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 29 MB/s | 16 MB 00:00 2024-03-18T01:08:34.896 INFO:teuthology.orchestra.run.smithi090.stdout:(10/12): libgfortran-11.4.1-3.el9.x86_64.rpm 1.1 MB/s | 803 kB 00:00 2024-03-18T01:08:34.897 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:00:48 ago on Mon 18 Mar 2024 01:07:46 AM UTC. 2024-03-18T01:08:34.921 INFO:teuthology.orchestra.run.smithi090.stdout:(11/12): libquadmath-11.4.1-3.el9.x86_64.rpm 490 kB/s | 195 kB 00:00 2024-03-18T01:08:35.063 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2024-03-18T01:08:35.065 INFO:teuthology.orchestra.run.smithi029.stdout:======================================================================================== 2024-03-18T01:08:35.065 INFO:teuthology.orchestra.run.smithi029.stdout: Package Arch Version Repository Size 2024-03-18T01:08:35.065 INFO:teuthology.orchestra.run.smithi029.stdout:======================================================================================== 2024-03-18T01:08:35.066 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2024-03-18T01:08:35.066 INFO:teuthology.orchestra.run.smithi029.stdout: ceph x86_64 2:17.2.7-684.g6716c9f3.el9 ceph 6.5 k 2024-03-18T01:08:35.066 INFO:teuthology.orchestra.run.smithi029.stdout:Installing dependencies: 2024-03-18T01:08:35.066 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mds x86_64 2:17.2.7-684.g6716c9f3.el9 ceph 2.1 M 2024-03-18T01:08:35.066 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr x86_64 2:17.2.7-684.g6716c9f3.el9 ceph 1.4 M 2024-03-18T01:08:35.066 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr-modules-core noarch 2:17.2.7-684.g6716c9f3.el9 ceph-noarch 239 k 2024-03-18T01:08:35.066 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mon x86_64 2:17.2.7-684.g6716c9f3.el9 ceph 3.7 M 2024-03-18T01:08:35.066 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-osd x86_64 2:17.2.7-684.g6716c9f3.el9 ceph 15 M 2024-03-18T01:08:35.066 INFO:teuthology.orchestra.run.smithi029.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-18T01:08:35.066 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-18T01:08:35.066 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-18T01:08:35.066 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-18T01:08:35.066 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-18T01:08:35.066 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-18T01:08:35.067 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-18T01:08:35.067 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-18T01:08:35.067 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-18T01:08:35.067 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-18T01:08:35.067 INFO:teuthology.orchestra.run.smithi029.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-18T01:08:35.067 INFO:teuthology.orchestra.run.smithi029.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-18T01:08:35.067 INFO:teuthology.orchestra.run.smithi029.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-18T01:08:35.067 INFO:teuthology.orchestra.run.smithi029.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-18T01:08:35.067 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-18T01:08:35.067 INFO:teuthology.orchestra.run.smithi029.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-18T01:08:35.067 INFO:teuthology.orchestra.run.smithi029.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-18T01:08:35.067 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-18T01:08:35.068 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-18T01:08:35.068 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-18T01:08:35.068 INFO:teuthology.orchestra.run.smithi029.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-18T01:08:35.068 INFO:teuthology.orchestra.run.smithi029.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-18T01:08:35.068 INFO:teuthology.orchestra.run.smithi029.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-18T01:08:35.068 INFO:teuthology.orchestra.run.smithi029.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-18T01:08:35.068 INFO:teuthology.orchestra.run.smithi029.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-18T01:08:35.068 INFO:teuthology.orchestra.run.smithi029.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-18T01:08:35.068 INFO:teuthology.orchestra.run.smithi029.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-18T01:08:35.068 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-18T01:08:35.068 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2024-03-18T01:08:35.068 INFO:teuthology.orchestra.run.smithi029.stdout:======================================================================================== 2024-03-18T01:08:35.068 INFO:teuthology.orchestra.run.smithi029.stdout:Install 33 Packages 2024-03-18T01:08:35.068 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-18T01:08:35.069 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 27 M 2024-03-18T01:08:35.069 INFO:teuthology.orchestra.run.smithi029.stdout:Installed size: 100 M 2024-03-18T01:08:35.069 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2024-03-18T01:08:35.589 INFO:teuthology.orchestra.run.smithi029.stdout:(1/33): ceph-17.2.7-684.g6716c9f3.el9.x86_64.rp 38 kB/s | 6.5 kB 00:00 2024-03-18T01:08:35.713 INFO:teuthology.orchestra.run.smithi090.stdout:(12/12): python3-devel-3.9.17-2.el9.x86_64.rpm 243 kB/s | 245 kB 00:01 2024-03-18T01:08:35.716 INFO:teuthology.orchestra.run.smithi090.stdout:-------------------------------------------------------------------------------- 2024-03-18T01:08:35.716 INFO:teuthology.orchestra.run.smithi090.stdout:Total 15 MB/s | 37 MB 00:02 2024-03-18T01:08:35.759 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction check 2024-03-18T01:08:35.785 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction check succeeded. 2024-03-18T01:08:35.785 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction test 2024-03-18T01:08:36.048 INFO:teuthology.orchestra.run.smithi029.stdout:(2/33): ceph-mon-17.2.7-684.g6716c9f3.el9.x86_6 8.1 MB/s | 3.7 MB 00:00 2024-03-18T01:08:36.107 INFO:teuthology.orchestra.run.smithi029.stdout:(3/33): ceph-mgr-17.2.7-684.g6716c9f3.el9.x86_6 2.0 MB/s | 1.4 MB 00:00 2024-03-18T01:08:36.215 INFO:teuthology.orchestra.run.smithi029.stdout:(4/33): ceph-mds-17.2.7-684.g6716c9f3.el9.x86_6 2.6 MB/s | 2.1 MB 00:00 2024-03-18T01:08:36.229 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction test succeeded. 2024-03-18T01:08:36.230 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction 2024-03-18T01:08:36.241 INFO:teuthology.orchestra.run.smithi029.stdout:(5/33): ceph-mgr-modules-core-17.2.7-684.g6716c 1.7 MB/s | 239 kB 00:00 2024-03-18T01:08:36.375 INFO:teuthology.orchestra.run.smithi029.stdout:(6/33): python3-markupsafe-1.1.1-12.el9.x86_64. 289 kB/s | 32 kB 00:00 2024-03-18T01:08:36.409 INFO:teuthology.orchestra.run.smithi029.stdout:(7/33): python3-mako-1.1.4-6.el9.noarch.rpm 747 kB/s | 144 kB 00:00 2024-03-18T01:08:36.434 INFO:teuthology.orchestra.run.smithi029.stdout:(8/33): python3-pytz-2021.1-5.el9.noarch.rpm 803 kB/s | 47 kB 00:00 2024-03-18T01:08:36.460 INFO:teuthology.orchestra.run.smithi029.stdout:(9/33): python3-toml-0.10.2-6.el9.noarch.rpm 764 kB/s | 39 kB 00:00 2024-03-18T01:08:36.493 INFO:teuthology.orchestra.run.smithi029.stdout:(10/33): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.0 MB/s | 241 kB 00:00 2024-03-18T01:08:36.578 INFO:teuthology.orchestra.run.smithi029.stdout:(11/33): python3-cryptography-36.0.1-4.el9.x86_ 9.7 MB/s | 1.1 MB 00:00 2024-03-18T01:08:36.603 INFO:teuthology.orchestra.run.smithi029.stdout:(12/33): python3-ply-3.11-14.el9.noarch.rpm 945 kB/s | 103 kB 00:00 2024-03-18T01:08:36.637 INFO:teuthology.orchestra.run.smithi029.stdout:(13/33): python3-pycparser-2.20-6.el9.noarch.rp 2.1 MB/s | 124 kB 00:00 2024-03-18T01:08:36.672 INFO:teuthology.orchestra.run.smithi029.stdout:(14/33): python3-urllib3-1.26.5-5.el9.noarch.rp 2.7 MB/s | 187 kB 00:00 2024-03-18T01:08:36.929 INFO:teuthology.orchestra.run.smithi090.stdout: Preparing : 1/1 2024-03-18T01:08:37.013 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-18T01:08:37.044 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-18T01:08:37.123 INFO:teuthology.orchestra.run.smithi029.stdout:(15/33): ceph-osd-17.2.7-684.g6716c9f3.el9.x86_ 14 MB/s | 15 MB 00:01 2024-03-18T01:08:37.140 INFO:teuthology.orchestra.run.smithi029.stdout:(16/33): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 93 kB/s | 43 kB 00:00 2024-03-18T01:08:37.141 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-18T01:08:37.174 INFO:teuthology.orchestra.run.smithi029.stdout:(17/33): python3-requests-2.25.1-8.el9.noarch.r 233 kB/s | 125 kB 00:00 2024-03-18T01:08:37.231 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-18T01:08:37.308 INFO:teuthology.orchestra.run.smithi029.stdout:(18/33): python3-cheroot-8.6.0-4.el9.noarch.rpm 931 kB/s | 172 kB 00:00 2024-03-18T01:08:37.325 INFO:teuthology.orchestra.run.smithi029.stdout:(19/33): python3-jaraco-8.2.1-3.el9.noarch.rpm 71 kB/s | 11 kB 00:00 2024-03-18T01:08:37.392 INFO:teuthology.orchestra.run.smithi029.stdout:(20/33): python3-jaraco-classes-3.2.1-5.el9.noa 210 kB/s | 18 kB 00:00 2024-03-18T01:08:37.402 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-18T01:08:37.430 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-18T01:08:37.468 INFO:teuthology.orchestra.run.smithi029.stdout:(21/33): python3-jaraco-collections-3.0.0-8.el9 162 kB/s | 23 kB 00:00 2024-03-18T01:08:37.494 INFO:teuthology.orchestra.run.smithi029.stdout:(22/33): python3-jaraco-functools-3.5.0-2.el9.n 193 kB/s | 19 kB 00:00 2024-03-18T01:08:37.515 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-18T01:08:37.569 INFO:teuthology.orchestra.run.smithi029.stdout:(23/33): python3-jaraco-text-3.2.0-6.el9.noarch 195 kB/s | 20 kB 00:00 2024-03-18T01:08:37.603 INFO:teuthology.orchestra.run.smithi029.stdout:(24/33): python3-logutils-0.3.5-21.el9.noarch.r 425 kB/s | 46 kB 00:00 2024-03-18T01:08:37.645 INFO:teuthology.orchestra.run.smithi029.stdout:(25/33): python3-cherrypy-18.6.1-2.el9.noarch.r 710 kB/s | 358 kB 00:00 2024-03-18T01:08:37.737 INFO:teuthology.orchestra.run.smithi029.stdout:(26/33): python3-portend-3.1.0-2.el9.noarch.rpm 178 kB/s | 16 kB 00:00 2024-03-18T01:08:37.771 INFO:teuthology.orchestra.run.smithi029.stdout:(27/33): python3-pecan-1.4.2-3.el9.noarch.rpm 1.6 MB/s | 272 kB 00:00 2024-03-18T01:08:37.797 INFO:teuthology.orchestra.run.smithi029.stdout:(28/33): python3-more-itertools-8.12.0-2.el9.no 346 kB/s | 79 kB 00:00 2024-03-18T01:08:37.840 INFO:teuthology.orchestra.run.smithi029.stdout:(29/33): python3-pyOpenSSL-21.0.0-1.el9.noarch. 873 kB/s | 90 kB 00:00 2024-03-18T01:08:37.866 INFO:teuthology.orchestra.run.smithi029.stdout:(30/33): python3-tempora-5.0.0-2.el9.noarch.rpm 379 kB/s | 36 kB 00:00 2024-03-18T01:08:37.950 INFO:teuthology.orchestra.run.smithi029.stdout:(31/33): python3-zc-lockfile-2.0-10.el9.noarch. 239 kB/s | 20 kB 00:00 2024-03-18T01:08:38.075 INFO:teuthology.orchestra.run.smithi029.stdout:(32/33): python3-webob-1.8.7-6.el9.noarch.rpm 826 kB/s | 230 kB 00:00 2024-03-18T01:08:38.142 INFO:teuthology.orchestra.run.smithi029.stdout:(33/33): python3-werkzeug-2.0.3-3.el9.1.noarch. 1.4 MB/s | 427 kB 00:00 2024-03-18T01:08:38.147 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2024-03-18T01:08:38.147 INFO:teuthology.orchestra.run.smithi029.stdout:Total 8.7 MB/s | 27 MB 00:03 2024-03-18T01:08:38.161 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2024-03-18T01:08:38.169 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-18T01:08:38.205 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2024-03-18T01:08:38.205 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2024-03-18T01:08:38.311 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-18T01:08:38.592 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2024-03-18T01:08:38.592 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2024-03-18T01:08:39.582 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-18T01:08:39.838 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2024-03-18T01:08:39.905 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/33 2024-03-18T01:08:39.947 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/33 2024-03-18T01:08:39.988 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/33 2024-03-18T01:08:40.084 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/33 2024-03-18T01:08:40.173 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/33 2024-03-18T01:08:40.228 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 6/33 2024-03-18T01:08:40.312 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/33 2024-03-18T01:08:40.373 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 8/33 2024-03-18T01:08:40.419 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 9/33 2024-03-18T01:08:40.478 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-ply-3.11-14.el9.noarch 10/33 2024-03-18T01:08:40.559 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 11/33 2024-03-18T01:08:40.636 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-18T01:08:40.729 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-mgr-diskprediction-local-2:17.2.7-684.g6716c9 12/12 2024-03-18T01:08:40.770 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 12/33 2024-03-18T01:08:40.828 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 13/33 2024-03-18T01:08:40.872 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 14/33 2024-03-18T01:08:40.910 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 15/33 2024-03-18T01:08:40.949 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 16/33 2024-03-18T01:08:41.031 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 17/33 2024-03-18T01:08:41.070 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 18/33 2024-03-18T01:08:41.104 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 19/33 2024-03-18T01:08:41.145 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 20/33 2024-03-18T01:08:41.189 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 21/33 2024-03-18T01:08:41.223 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 22/33 2024-03-18T01:08:41.319 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 23/33 2024-03-18T01:08:41.357 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 24/33 2024-03-18T01:08:41.442 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 25/33 2024-03-18T01:08:41.594 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 26/33 2024-03-18T01:08:41.736 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 27/33 2024-03-18T01:08:41.902 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-mgr-modules-core-2:17.2.7-684.g6716c9f3.el9.n 28/33 2024-03-18T01:08:41.935 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-mgr-2:17.2.7-684.g6716c9f3.el9.x86_64 29/33 2024-03-18T01:08:41.959 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-mgr-2:17.2.7-684.g6716c9f3.el9.x86_64 29/33 2024-03-18T01:08:41.959 INFO:teuthology.orchestra.run.smithi029.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-18T01:08:41.959 INFO:teuthology.orchestra.run.smithi029.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-18T01:08:41.960 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-18T01:08:41.960 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-18T01:08:41.960 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-18T01:08:43.387 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-osd-2:17.2.7-684.g6716c9f3.el9.x86_64 30/33 2024-03-18T01:08:43.413 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-osd-2:17.2.7-684.g6716c9f3.el9.x86_64 30/33 2024-03-18T01:08:43.413 INFO:teuthology.orchestra.run.smithi029.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-18T01:08:43.413 INFO:teuthology.orchestra.run.smithi029.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-18T01:08:43.413 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-18T01:08:43.414 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-18T01:08:43.414 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-18T01:08:43.822 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-mon-2:17.2.7-684.g6716c9f3.el9.x86_64 31/33 2024-03-18T01:08:43.844 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-mon-2:17.2.7-684.g6716c9f3.el9.x86_64 31/33 2024-03-18T01:08:43.845 INFO:teuthology.orchestra.run.smithi029.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-18T01:08:43.845 INFO:teuthology.orchestra.run.smithi029.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-18T01:08:43.845 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-18T01:08:43.845 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-18T01:08:43.845 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-18T01:08:43.869 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.2.7-684.g6716c9 12/12 2024-03-18T01:08:43.869 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.2.7-684.g6716c9 1/12 2024-03-18T01:08:43.869 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-18T01:08:43.869 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-18T01:08:43.869 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-18T01:08:43.869 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-18T01:08:43.869 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-18T01:08:43.869 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-18T01:08:43.869 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-18T01:08:43.869 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-18T01:08:43.870 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-18T01:08:43.870 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-18T01:08:44.058 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-mds-2:17.2.7-684.g6716c9f3.el9.x86_64 32/33 2024-03-18T01:08:44.081 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-mds-2:17.2.7-684.g6716c9f3.el9.x86_64 32/33 2024-03-18T01:08:44.081 INFO:teuthology.orchestra.run.smithi029.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-18T01:08:44.081 INFO:teuthology.orchestra.run.smithi029.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-18T01:08:44.081 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-18T01:08:44.081 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-18T01:08:44.081 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-18T01:08:44.150 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-2:17.2.7-684.g6716c9f3.el9.x86_64 33/33 2024-03-18T01:08:44.281 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-18T01:08:44.281 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-18T01:08:44.281 INFO:teuthology.orchestra.run.smithi090.stdout:Installed: 2024-03-18T01:08:44.282 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mgr-diskprediction-local-2:17.2.7-684.g6716c9f3.el9.noarch 2024-03-18T01:08:44.282 INFO:teuthology.orchestra.run.smithi090.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-18T01:08:44.282 INFO:teuthology.orchestra.run.smithi090.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-18T01:08:44.282 INFO:teuthology.orchestra.run.smithi090.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-18T01:08:44.282 INFO:teuthology.orchestra.run.smithi090.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-18T01:08:44.282 INFO:teuthology.orchestra.run.smithi090.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-18T01:08:44.282 INFO:teuthology.orchestra.run.smithi090.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-18T01:08:44.282 INFO:teuthology.orchestra.run.smithi090.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-18T01:08:44.282 INFO:teuthology.orchestra.run.smithi090.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-18T01:08:44.282 INFO:teuthology.orchestra.run.smithi090.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-18T01:08:44.282 INFO:teuthology.orchestra.run.smithi090.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-18T01:08:44.282 INFO:teuthology.orchestra.run.smithi090.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-18T01:08:44.282 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-18T01:08:44.283 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-03-18T01:08:44.737 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install ceph-mgr-rook 2024-03-18T01:08:45.310 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:01:18 ago on Mon 18 Mar 2024 01:07:27 AM UTC. 2024-03-18T01:08:45.459 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-03-18T01:08:45.461 INFO:teuthology.orchestra.run.smithi090.stdout:======================================================================================= 2024-03-18T01:08:45.461 INFO:teuthology.orchestra.run.smithi090.stdout: Package Arch Version Repository Size 2024-03-18T01:08:45.461 INFO:teuthology.orchestra.run.smithi090.stdout:======================================================================================= 2024-03-18T01:08:45.461 INFO:teuthology.orchestra.run.smithi090.stdout:Installing: 2024-03-18T01:08:45.461 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mgr-rook noarch 2:17.2.7-684.g6716c9f3.el9 ceph-noarch 47 k 2024-03-18T01:08:45.461 INFO:teuthology.orchestra.run.smithi090.stdout:Installing dependencies: 2024-03-18T01:08:45.461 INFO:teuthology.orchestra.run.smithi090.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-18T01:08:45.461 INFO:teuthology.orchestra.run.smithi090.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-18T01:08:45.461 INFO:teuthology.orchestra.run.smithi090.stdout: python3-google-auth noarch 1:2.28.0-1.el9 epel 203 k 2024-03-18T01:08:45.461 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-18T01:08:45.461 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-18T01:08:45.461 INFO:teuthology.orchestra.run.smithi090.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-18T01:08:45.462 INFO:teuthology.orchestra.run.smithi090.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-18T01:08:45.462 INFO:teuthology.orchestra.run.smithi090.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-18T01:08:45.462 INFO:teuthology.orchestra.run.smithi090.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-18T01:08:45.462 INFO:teuthology.orchestra.run.smithi090.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-18T01:08:45.462 INFO:teuthology.orchestra.run.smithi090.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-18T01:08:45.462 INFO:teuthology.orchestra.run.smithi090.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-18T01:08:45.462 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-18T01:08:45.462 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction Summary 2024-03-18T01:08:45.462 INFO:teuthology.orchestra.run.smithi090.stdout:======================================================================================= 2024-03-18T01:08:45.462 INFO:teuthology.orchestra.run.smithi090.stdout:Install 13 Packages 2024-03-18T01:08:45.462 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-18T01:08:45.463 INFO:teuthology.orchestra.run.smithi090.stdout:Total download size: 2.0 M 2024-03-18T01:08:45.463 INFO:teuthology.orchestra.run.smithi090.stdout:Installed size: 26 M 2024-03-18T01:08:45.463 INFO:teuthology.orchestra.run.smithi090.stdout:Downloading Packages: 2024-03-18T01:08:46.632 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-2:17.2.7-684.g6716c9f3.el9.x86_64 33/33 2024-03-18T01:08:46.633 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-2:17.2.7-684.g6716c9f3.el9.x86_64 1/33 2024-03-18T01:08:46.633 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-mds-2:17.2.7-684.g6716c9f3.el9.x86_64 2/33 2024-03-18T01:08:46.633 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-mgr-2:17.2.7-684.g6716c9f3.el9.x86_64 3/33 2024-03-18T01:08:46.633 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-mon-2:17.2.7-684.g6716c9f3.el9.x86_64 4/33 2024-03-18T01:08:46.633 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-osd-2:17.2.7-684.g6716c9f3.el9.x86_64 5/33 2024-03-18T01:08:46.633 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-mgr-modules-core-2:17.2.7-684.g6716c9f3.el9.n 6/33 2024-03-18T01:08:46.633 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 7/33 2024-03-18T01:08:46.633 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 8/33 2024-03-18T01:08:46.633 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 9/33 2024-03-18T01:08:46.633 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 10/33 2024-03-18T01:08:46.633 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 11/33 2024-03-18T01:08:46.633 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 12/33 2024-03-18T01:08:46.633 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-ply-3.11-14.el9.noarch 13/33 2024-03-18T01:08:46.633 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 14/33 2024-03-18T01:08:46.634 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 15/33 2024-03-18T01:08:46.634 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 16/33 2024-03-18T01:08:46.634 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 17/33 2024-03-18T01:08:46.634 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 18/33 2024-03-18T01:08:46.634 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 19/33 2024-03-18T01:08:46.634 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 20/33 2024-03-18T01:08:46.634 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 21/33 2024-03-18T01:08:46.634 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 22/33 2024-03-18T01:08:46.634 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 23/33 2024-03-18T01:08:46.634 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 24/33 2024-03-18T01:08:46.634 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 25/33 2024-03-18T01:08:46.634 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 26/33 2024-03-18T01:08:46.634 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 27/33 2024-03-18T01:08:46.634 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 28/33 2024-03-18T01:08:46.635 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 29/33 2024-03-18T01:08:46.635 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 30/33 2024-03-18T01:08:46.635 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 31/33 2024-03-18T01:08:46.635 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 32/33 2024-03-18T01:08:47.648 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 33/33 2024-03-18T01:08:47.649 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-18T01:08:47.649 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2024-03-18T01:08:47.649 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-2:17.2.7-684.g6716c9f3.el9.x86_64 2024-03-18T01:08:47.649 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mds-2:17.2.7-684.g6716c9f3.el9.x86_64 2024-03-18T01:08:47.649 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr-2:17.2.7-684.g6716c9f3.el9.x86_64 2024-03-18T01:08:47.649 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr-modules-core-2:17.2.7-684.g6716c9f3.el9.noarch 2024-03-18T01:08:47.649 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mon-2:17.2.7-684.g6716c9f3.el9.x86_64 2024-03-18T01:08:47.649 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-osd-2:17.2.7-684.g6716c9f3.el9.x86_64 2024-03-18T01:08:47.650 INFO:teuthology.orchestra.run.smithi029.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-18T01:08:47.650 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-18T01:08:47.650 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-18T01:08:47.650 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-18T01:08:47.650 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-18T01:08:47.650 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-18T01:08:47.650 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-18T01:08:47.650 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-18T01:08:47.650 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-18T01:08:47.651 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-18T01:08:47.651 INFO:teuthology.orchestra.run.smithi029.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-18T01:08:47.651 INFO:teuthology.orchestra.run.smithi029.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-18T01:08:47.651 INFO:teuthology.orchestra.run.smithi029.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-18T01:08:47.651 INFO:teuthology.orchestra.run.smithi029.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-18T01:08:47.651 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-18T01:08:47.651 INFO:teuthology.orchestra.run.smithi029.stdout: python3-ply-3.11-14.el9.noarch 2024-03-18T01:08:47.651 INFO:teuthology.orchestra.run.smithi029.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-18T01:08:47.651 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-18T01:08:47.651 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-18T01:08:47.652 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-18T01:08:47.652 INFO:teuthology.orchestra.run.smithi029.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-18T01:08:47.652 INFO:teuthology.orchestra.run.smithi029.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-18T01:08:47.652 INFO:teuthology.orchestra.run.smithi029.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-18T01:08:47.652 INFO:teuthology.orchestra.run.smithi029.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-18T01:08:47.652 INFO:teuthology.orchestra.run.smithi029.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-18T01:08:47.652 INFO:teuthology.orchestra.run.smithi029.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-18T01:08:47.652 INFO:teuthology.orchestra.run.smithi029.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-18T01:08:47.652 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-18T01:08:47.652 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2024-03-18T01:08:48.134 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install ceph-base 2024-03-18T01:08:48.708 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:01:02 ago on Mon 18 Mar 2024 01:07:46 AM UTC. 2024-03-18T01:08:48.808 INFO:teuthology.orchestra.run.smithi029.stdout:Package ceph-base-2:17.2.7-684.g6716c9f3.el9.x86_64 is already installed. 2024-03-18T01:08:48.864 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2024-03-18T01:08:48.865 INFO:teuthology.orchestra.run.smithi029.stdout:Nothing to do. 2024-03-18T01:08:48.865 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2024-03-18T01:08:48.938 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install cephadm 2024-03-18T01:08:49.506 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:01:03 ago on Mon 18 Mar 2024 01:07:46 AM UTC. 2024-03-18T01:08:49.663 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2024-03-18T01:08:49.663 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-03-18T01:08:49.663 INFO:teuthology.orchestra.run.smithi029.stdout: Package Arch Version Repository Size 2024-03-18T01:08:49.664 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-03-18T01:08:49.664 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2024-03-18T01:08:49.664 INFO:teuthology.orchestra.run.smithi029.stdout: cephadm noarch 2:17.2.7-684.g6716c9f3.el9 ceph-noarch 82 k 2024-03-18T01:08:49.664 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-18T01:08:49.664 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2024-03-18T01:08:49.664 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-03-18T01:08:49.664 INFO:teuthology.orchestra.run.smithi029.stdout:Install 1 Package 2024-03-18T01:08:49.664 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-18T01:08:49.664 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 82 k 2024-03-18T01:08:49.664 INFO:teuthology.orchestra.run.smithi029.stdout:Installed size: 372 k 2024-03-18T01:08:49.664 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2024-03-18T01:08:49.877 INFO:teuthology.orchestra.run.smithi029.stdout:cephadm-17.2.7-684.g6716c9f3.el9.noarch.rpm 386 kB/s | 82 kB 00:00 2024-03-18T01:08:49.877 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2024-03-18T01:08:49.878 INFO:teuthology.orchestra.run.smithi029.stdout:Total 384 kB/s | 82 kB 00:00 2024-03-18T01:08:49.878 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2024-03-18T01:08:49.883 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2024-03-18T01:08:49.883 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2024-03-18T01:08:49.885 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2024-03-18T01:08:49.885 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2024-03-18T01:08:49.965 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2024-03-18T01:08:50.414 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: cephadm-2:17.2.7-684.g6716c9f3.el9.noarch 1/1 2024-03-18T01:08:50.515 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : cephadm-2:17.2.7-684.g6716c9f3.el9.noarch 1/1 2024-03-18T01:08:50.973 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: cephadm-2:17.2.7-684.g6716c9f3.el9.noarch 1/1 2024-03-18T01:08:51.289 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : cephadm-2:17.2.7-684.g6716c9f3.el9.noarch 1/1 2024-03-18T01:08:51.289 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-18T01:08:51.289 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2024-03-18T01:08:51.289 INFO:teuthology.orchestra.run.smithi029.stdout: cephadm-2:17.2.7-684.g6716c9f3.el9.noarch 2024-03-18T01:08:51.290 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-18T01:08:51.290 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2024-03-18T01:08:51.536 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install ceph-immutable-object-cache 2024-03-18T01:08:52.113 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:01:06 ago on Mon 18 Mar 2024 01:07:46 AM UTC. 2024-03-18T01:08:52.269 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2024-03-18T01:08:52.270 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-03-18T01:08:52.270 INFO:teuthology.orchestra.run.smithi029.stdout: Package Arch Version Repo Size 2024-03-18T01:08:52.270 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-03-18T01:08:52.271 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2024-03-18T01:08:52.271 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-immutable-object-cache x86_64 2:17.2.7-684.g6716c9f3.el9 ceph 137 k 2024-03-18T01:08:52.271 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-18T01:08:52.271 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2024-03-18T01:08:52.271 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-03-18T01:08:52.271 INFO:teuthology.orchestra.run.smithi029.stdout:Install 1 Package 2024-03-18T01:08:52.271 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-18T01:08:52.271 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 137 k 2024-03-18T01:08:52.271 INFO:teuthology.orchestra.run.smithi029.stdout:Installed size: 379 k 2024-03-18T01:08:52.271 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2024-03-18T01:08:52.485 INFO:teuthology.orchestra.run.smithi029.stdout:ceph-immutable-object-cache-17.2.7-684.g6716c9f 643 kB/s | 137 kB 00:00 2024-03-18T01:08:52.485 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2024-03-18T01:08:52.486 INFO:teuthology.orchestra.run.smithi029.stdout:Total 639 kB/s | 137 kB 00:00 2024-03-18T01:08:52.486 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2024-03-18T01:08:52.493 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2024-03-18T01:08:52.494 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2024-03-18T01:08:52.547 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2024-03-18T01:08:52.547 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2024-03-18T01:08:52.685 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2024-03-18T01:08:52.754 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-immutable-object-cache-2:17.2.7-684.g6716c9f3.e 1/1 2024-03-18T01:08:52.779 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.2.7-684.g6716c9f3.e 1/1 2024-03-18T01:08:52.779 INFO:teuthology.orchestra.run.smithi029.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-18T01:08:52.779 INFO:teuthology.orchestra.run.smithi029.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-18T01:08:52.779 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-18T01:08:53.933 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-immutable-object-cache-2:17.2.7-684.g6716c9f3.e 1/1 2024-03-18T01:08:53.933 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-18T01:08:53.933 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2024-03-18T01:08:53.933 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-immutable-object-cache-2:17.2.7-684.g6716c9f3.el9.x86_64 2024-03-18T01:08:53.933 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-18T01:08:53.934 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2024-03-18T01:08:54.209 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install ceph-mgr 2024-03-18T01:08:54.778 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:01:08 ago on Mon 18 Mar 2024 01:07:46 AM UTC. 2024-03-18T01:08:54.878 INFO:teuthology.orchestra.run.smithi029.stdout:Package ceph-mgr-2:17.2.7-684.g6716c9f3.el9.x86_64 is already installed. 2024-03-18T01:08:54.934 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2024-03-18T01:08:54.935 INFO:teuthology.orchestra.run.smithi029.stdout:Nothing to do. 2024-03-18T01:08:54.935 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2024-03-18T01:08:55.015 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install ceph-mgr-dashboard 2024-03-18T01:08:55.581 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:01:09 ago on Mon 18 Mar 2024 01:07:46 AM UTC. 2024-03-18T01:08:55.736 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2024-03-18T01:08:55.737 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-03-18T01:08:55.737 INFO:teuthology.orchestra.run.smithi029.stdout: Package Arch Version Repository Size 2024-03-18T01:08:55.737 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-03-18T01:08:55.737 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2024-03-18T01:08:55.737 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr-dashboard noarch 2:17.2.7-684.g6716c9f3.el9 ceph-noarch 1.7 M 2024-03-18T01:08:55.737 INFO:teuthology.orchestra.run.smithi029.stdout:Installing dependencies: 2024-03-18T01:08:55.737 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-grafana-dashboards noarch 2:17.2.7-684.g6716c9f3.el9 ceph-noarch 23 k 2024-03-18T01:08:55.737 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-prometheus-alerts noarch 2:17.2.7-684.g6716c9f3.el9 ceph-noarch 15 k 2024-03-18T01:08:55.737 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2024-03-18T01:08:55.737 INFO:teuthology.orchestra.run.smithi029.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-18T01:08:55.737 INFO:teuthology.orchestra.run.smithi029.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-18T01:08:55.737 INFO:teuthology.orchestra.run.smithi029.stdout:Installing weak dependencies: 2024-03-18T01:08:55.738 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2024-03-18T01:08:55.738 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-18T01:08:55.738 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2024-03-18T01:08:55.738 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-03-18T01:08:55.738 INFO:teuthology.orchestra.run.smithi029.stdout:Install 7 Packages 2024-03-18T01:08:55.738 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-18T01:08:55.738 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 2.0 M 2024-03-18T01:08:55.739 INFO:teuthology.orchestra.run.smithi029.stdout:Installed size: 8.1 M 2024-03-18T01:08:55.739 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2024-03-18T01:08:56.314 INFO:teuthology.orchestra.run.smithi029.stdout:(1/7): ceph-prometheus-alerts-17.2.7-684.g6716c 75 kB/s | 15 kB 00:00 2024-03-18T01:08:56.340 INFO:teuthology.orchestra.run.smithi029.stdout:(2/7): ceph-grafana-dashboards-17.2.7-684.g6716 106 kB/s | 23 kB 00:00 2024-03-18T01:08:56.541 INFO:teuthology.orchestra.run.smithi029.stdout:(3/7): python3-jwt+crypto-2.4.0-1.el9.noarch.rp 40 kB/s | 9.0 kB 00:00 2024-03-18T01:08:56.600 INFO:teuthology.orchestra.run.smithi029.stdout:(4/7): ceph-mgr-dashboard-17.2.7-684.g6716c9f3. 3.5 MB/s | 1.7 MB 00:00 2024-03-18T01:08:56.625 INFO:teuthology.orchestra.run.smithi029.stdout:(5/7): python3-jwt-2.4.0-1.el9.noarch.rpm 144 kB/s | 41 kB 00:00 2024-03-18T01:08:56.650 INFO:teuthology.orchestra.run.smithi029.stdout:(6/7): python3-repoze-lru-0.7-16.el9.noarch.rpm 281 kB/s | 31 kB 00:00 2024-03-18T01:08:56.960 INFO:teuthology.orchestra.run.smithi029.stdout:(7/7): python3-routes-2.5.1-5.el9.noarch.rpm 523 kB/s | 188 kB 00:00 2024-03-18T01:08:56.964 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2024-03-18T01:08:56.964 INFO:teuthology.orchestra.run.smithi029.stdout:Total 1.6 MB/s | 2.0 MB 00:01 2024-03-18T01:08:56.965 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2024-03-18T01:08:56.973 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2024-03-18T01:08:56.973 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2024-03-18T01:08:57.039 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2024-03-18T01:08:57.040 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2024-03-18T01:08:57.351 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2024-03-18T01:08:57.438 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 1/7 2024-03-18T01:08:57.473 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 2/7 2024-03-18T01:08:57.521 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/7 2024-03-18T01:08:57.559 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/7 2024-03-18T01:08:57.597 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-prometheus-alerts-2:17.2.7-684.g6716c9f3.el9.no 5/7 2024-03-18T01:08:57.842 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-grafana-dashboards-2:17.2.7-684.g6716c9f3.el9.n 6/7 2024-03-18T01:08:57.876 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-mgr-dashboard-2:17.2.7-684.g6716c9f3.el9.noarch 7/7 2024-03-18T01:08:58.549 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.2.7-684.g6716c9f3.el9.noarch 7/7 2024-03-18T01:08:58.549 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-grafana-dashboards-2:17.2.7-684.g6716c9f3.el9.n 1/7 2024-03-18T01:08:58.550 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-mgr-dashboard-2:17.2.7-684.g6716c9f3.el9.noarch 2/7 2024-03-18T01:08:58.550 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-prometheus-alerts-2:17.2.7-684.g6716c9f3.el9.no 3/7 2024-03-18T01:08:58.550 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 4/7 2024-03-18T01:08:58.550 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 5/7 2024-03-18T01:08:58.550 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 6/7 2024-03-18T01:08:58.865 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 7/7 2024-03-18T01:08:58.865 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-18T01:08:58.865 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2024-03-18T01:08:58.865 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-grafana-dashboards-2:17.2.7-684.g6716c9f3.el9.noarch 2024-03-18T01:08:58.865 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr-dashboard-2:17.2.7-684.g6716c9f3.el9.noarch 2024-03-18T01:08:58.865 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-prometheus-alerts-2:17.2.7-684.g6716c9f3.el9.noarch 2024-03-18T01:08:58.865 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jwt-2.4.0-1.el9.noarch 2024-03-18T01:08:58.865 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2024-03-18T01:08:58.866 INFO:teuthology.orchestra.run.smithi029.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-18T01:08:58.866 INFO:teuthology.orchestra.run.smithi029.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-18T01:08:58.866 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-18T01:08:58.866 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2024-03-18T01:08:59.148 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-18T01:08:59.717 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:01:13 ago on Mon 18 Mar 2024 01:07:46 AM UTC. 2024-03-18T01:08:59.876 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2024-03-18T01:08:59.877 INFO:teuthology.orchestra.run.smithi029.stdout:=========================================================================================== 2024-03-18T01:08:59.877 INFO:teuthology.orchestra.run.smithi029.stdout: Package Arch Version Repository Size 2024-03-18T01:08:59.878 INFO:teuthology.orchestra.run.smithi029.stdout:=========================================================================================== 2024-03-18T01:08:59.878 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2024-03-18T01:08:59.878 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr-diskprediction-local noarch 2:17.2.7-684.g6716c9f3.el9 ceph-noarch 7.4 M 2024-03-18T01:08:59.878 INFO:teuthology.orchestra.run.smithi029.stdout:Installing dependencies: 2024-03-18T01:08:59.878 INFO:teuthology.orchestra.run.smithi029.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-18T01:08:59.878 INFO:teuthology.orchestra.run.smithi029.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-18T01:08:59.878 INFO:teuthology.orchestra.run.smithi029.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-18T01:08:59.878 INFO:teuthology.orchestra.run.smithi029.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-18T01:08:59.878 INFO:teuthology.orchestra.run.smithi029.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-18T01:08:59.878 INFO:teuthology.orchestra.run.smithi029.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-18T01:08:59.878 INFO:teuthology.orchestra.run.smithi029.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-18T01:08:59.878 INFO:teuthology.orchestra.run.smithi029.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-18T01:08:59.878 INFO:teuthology.orchestra.run.smithi029.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-18T01:08:59.879 INFO:teuthology.orchestra.run.smithi029.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-18T01:08:59.879 INFO:teuthology.orchestra.run.smithi029.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-18T01:08:59.879 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-18T01:08:59.879 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2024-03-18T01:08:59.879 INFO:teuthology.orchestra.run.smithi029.stdout:=========================================================================================== 2024-03-18T01:08:59.879 INFO:teuthology.orchestra.run.smithi029.stdout:Install 12 Packages 2024-03-18T01:08:59.879 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-18T01:08:59.879 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 37 M 2024-03-18T01:08:59.880 INFO:teuthology.orchestra.run.smithi029.stdout:Installed size: 209 M 2024-03-18T01:08:59.880 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2024-03-18T01:09:00.593 INFO:teuthology.orchestra.run.smithi029.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 210 kB/s | 32 kB 00:00 2024-03-18T01:09:00.635 INFO:teuthology.orchestra.run.smithi029.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 443 kB/s | 17 kB 00:00 2024-03-18T01:09:00.710 INFO:teuthology.orchestra.run.smithi029.stdout:(3/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 11 MB/s | 3.0 MB 00:00 2024-03-18T01:09:00.736 INFO:teuthology.orchestra.run.smithi029.stdout:(4/12): openblas-0.3.21-2.el9.x86_64.rpm 339 kB/s | 34 kB 00:00 2024-03-18T01:09:00.930 INFO:teuthology.orchestra.run.smithi029.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 22 MB/s | 4.7 MB 00:00 2024-03-18T01:09:01.030 INFO:teuthology.orchestra.run.smithi029.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 16 MB/s | 4.8 MB 00:00 2024-03-18T01:09:01.056 INFO:teuthology.orchestra.run.smithi029.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 1.7 MB/s | 217 kB 00:00 2024-03-18T01:09:01.465 INFO:teuthology.orchestra.run.smithi029.stdout:(8/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 36 MB/s | 16 MB 00:00 2024-03-18T01:09:01.515 INFO:teuthology.orchestra.run.smithi029.stdout:(9/12): libgfortran-11.4.1-3.el9.x86_64.rpm 1.7 MB/s | 803 kB 00:00 2024-03-18T01:09:01.607 INFO:teuthology.orchestra.run.smithi029.stdout:(10/12): libquadmath-11.4.1-3.el9.x86_64.rpm 1.3 MB/s | 195 kB 00:00 2024-03-18T01:09:02.266 INFO:teuthology.orchestra.run.smithi029.stdout:(11/12): ceph-mgr-diskprediction-local-17.2.7-6 4.1 MB/s | 7.4 MB 00:01 2024-03-18T01:09:02.575 INFO:teuthology.orchestra.run.smithi029.stdout:(12/12): python3-devel-3.9.17-2.el9.x86_64.rpm 233 kB/s | 245 kB 00:01 2024-03-18T01:09:02.577 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2024-03-18T01:09:02.577 INFO:teuthology.orchestra.run.smithi029.stdout:Total 14 MB/s | 37 MB 00:02 2024-03-18T01:09:02.613 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2024-03-18T01:09:02.640 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2024-03-18T01:09:02.641 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2024-03-18T01:09:03.110 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2024-03-18T01:09:03.111 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2024-03-18T01:09:03.830 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2024-03-18T01:09:03.922 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-18T01:09:03.956 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-18T01:09:04.054 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-18T01:09:04.095 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-18T01:09:04.291 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-18T01:09:04.320 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-18T01:09:04.408 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-18T01:09:05.108 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-18T01:09:05.235 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-18T01:09:06.322 INFO:teuthology.orchestra.run.smithi090.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 169 kB/s | 24 kB 00:00 2024-03-18T01:09:06.347 INFO:teuthology.orchestra.run.smithi090.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 107 kB/s | 18 kB 00:00 2024-03-18T01:09:06.464 INFO:teuthology.orchestra.run.smithi090.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 1.2 MB/s | 174 kB 00:00 2024-03-18T01:09:06.489 INFO:teuthology.orchestra.run.smithi090.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 932 kB/s | 132 kB 00:00 2024-03-18T01:09:06.515 INFO:teuthology.orchestra.run.smithi090.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2024-03-18T01:09:06.540 INFO:teuthology.orchestra.run.smithi090.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 857 kB/s | 43 kB 00:00 2024-03-18T01:09:06.546 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-18T01:09:06.565 INFO:teuthology.orchestra.run.smithi090.stdout:(7/13): ceph-mgr-rook-17.2.7-684.g6716c9f3.el9. 122 kB/s | 47 kB 00:00 2024-03-18T01:09:06.633 INFO:teuthology.orchestra.run.smithi090.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 163 kB/s | 15 kB 00:00 2024-03-18T01:09:06.658 INFO:teuthology.orchestra.run.smithi090.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 226 kB/s | 32 kB 00:00 2024-03-18T01:09:06.733 INFO:teuthology.orchestra.run.smithi090.stdout:(10/13): python3-google-auth-2.28.0-1.el9.noarc 1.2 MB/s | 203 kB 00:00 2024-03-18T01:09:06.758 INFO:teuthology.orchestra.run.smithi090.stdout:(11/13): python3-rsa-4.9-2.el9.noarch.rpm 589 kB/s | 59 kB 00:00 2024-03-18T01:09:06.792 INFO:teuthology.orchestra.run.smithi090.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 1.5 MB/s | 90 kB 00:00 2024-03-18T01:09:06.859 INFO:teuthology.orchestra.run.smithi090.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 4.5 MB/s | 1.0 MB 00:00 2024-03-18T01:09:06.861 INFO:teuthology.orchestra.run.smithi090.stdout:-------------------------------------------------------------------------------- 2024-03-18T01:09:06.861 INFO:teuthology.orchestra.run.smithi090.stdout:Total 98 kB/s | 2.0 MB 00:21 2024-03-18T01:09:06.862 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction check 2024-03-18T01:09:06.877 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction check succeeded. 2024-03-18T01:09:06.877 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction test 2024-03-18T01:09:06.981 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction test succeeded. 2024-03-18T01:09:06.982 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction 2024-03-18T01:09:07.503 INFO:teuthology.orchestra.run.smithi090.stdout: Preparing : 1/1 2024-03-18T01:09:07.644 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-18T01:09:07.676 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-mgr-diskprediction-local-2:17.2.7-684.g6716c9 12/12 2024-03-18T01:09:07.681 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-18T01:09:07.744 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-18T01:09:07.799 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-18T01:09:07.838 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-18T01:09:07.880 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-18T01:09:07.973 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-18T01:09:08.083 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-google-auth-1:2.28.0-1.el9.noarch 7/13 2024-03-18T01:09:08.140 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-18T01:09:08.863 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-18T01:09:08.948 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-18T01:09:08.973 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-18T01:09:09.019 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-18T01:09:09.051 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-mgr-rook-2:17.2.7-684.g6716c9f3.el9.noarch 13/13 2024-03-18T01:09:09.970 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-mgr-rook-2:17.2.7-684.g6716c9f3.el9.noarch 13/13 2024-03-18T01:09:09.971 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-mgr-rook-2:17.2.7-684.g6716c9f3.el9.noarch 1/13 2024-03-18T01:09:09.971 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-18T01:09:09.971 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-18T01:09:09.971 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-18T01:09:09.971 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-18T01:09:09.971 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-18T01:09:09.971 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-18T01:09:09.971 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-18T01:09:09.971 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-18T01:09:09.971 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-google-auth-1:2.28.0-1.el9.noarch 10/13 2024-03-18T01:09:09.971 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-18T01:09:09.972 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-18T01:09:10.334 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-18T01:09:10.334 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-18T01:09:10.334 INFO:teuthology.orchestra.run.smithi090.stdout:Installed: 2024-03-18T01:09:10.334 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mgr-rook-2:17.2.7-684.g6716c9f3.el9.noarch 2024-03-18T01:09:10.334 INFO:teuthology.orchestra.run.smithi090.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-18T01:09:10.334 INFO:teuthology.orchestra.run.smithi090.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-18T01:09:10.334 INFO:teuthology.orchestra.run.smithi090.stdout: python3-google-auth-1:2.28.0-1.el9.noarch 2024-03-18T01:09:10.335 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-18T01:09:10.335 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-18T01:09:10.335 INFO:teuthology.orchestra.run.smithi090.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-18T01:09:10.335 INFO:teuthology.orchestra.run.smithi090.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-18T01:09:10.335 INFO:teuthology.orchestra.run.smithi090.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-18T01:09:10.335 INFO:teuthology.orchestra.run.smithi090.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-18T01:09:10.335 INFO:teuthology.orchestra.run.smithi090.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-18T01:09:10.335 INFO:teuthology.orchestra.run.smithi090.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-18T01:09:10.335 INFO:teuthology.orchestra.run.smithi090.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-18T01:09:10.335 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-18T01:09:10.335 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-03-18T01:09:10.659 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install ceph-mgr-cephadm 2024-03-18T01:09:11.223 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:01:44 ago on Mon 18 Mar 2024 01:07:27 AM UTC. 2024-03-18T01:09:11.371 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-03-18T01:09:11.372 INFO:teuthology.orchestra.run.smithi090.stdout:======================================================================================= 2024-03-18T01:09:11.373 INFO:teuthology.orchestra.run.smithi090.stdout: Package Arch Version Repository Size 2024-03-18T01:09:11.373 INFO:teuthology.orchestra.run.smithi090.stdout:======================================================================================= 2024-03-18T01:09:11.373 INFO:teuthology.orchestra.run.smithi090.stdout:Installing: 2024-03-18T01:09:11.373 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mgr-cephadm noarch 2:17.2.7-684.g6716c9f3.el9 ceph-noarch 123 k 2024-03-18T01:09:11.373 INFO:teuthology.orchestra.run.smithi090.stdout:Installing dependencies: 2024-03-18T01:09:11.373 INFO:teuthology.orchestra.run.smithi090.stdout: python3-asyncssh noarch 2.13.2-4.el9 epel 548 k 2024-03-18T01:09:11.373 INFO:teuthology.orchestra.run.smithi090.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-18T01:09:11.373 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-18T01:09:11.373 INFO:teuthology.orchestra.run.smithi090.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-18T01:09:11.373 INFO:teuthology.orchestra.run.smithi090.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-18T01:09:11.373 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-18T01:09:11.373 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction Summary 2024-03-18T01:09:11.374 INFO:teuthology.orchestra.run.smithi090.stdout:======================================================================================= 2024-03-18T01:09:11.374 INFO:teuthology.orchestra.run.smithi090.stdout:Install 6 Packages 2024-03-18T01:09:11.374 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-18T01:09:11.374 INFO:teuthology.orchestra.run.smithi090.stdout:Total download size: 6.8 M 2024-03-18T01:09:11.374 INFO:teuthology.orchestra.run.smithi090.stdout:Installed size: 33 M 2024-03-18T01:09:11.374 INFO:teuthology.orchestra.run.smithi090.stdout:Downloading Packages: 2024-03-18T01:09:11.798 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.2.7-684.g6716c9 12/12 2024-03-18T01:09:11.799 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.2.7-684.g6716c9 1/12 2024-03-18T01:09:11.799 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-18T01:09:11.799 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-18T01:09:11.799 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-18T01:09:11.799 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-18T01:09:11.799 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-18T01:09:11.799 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-18T01:09:11.799 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-18T01:09:11.799 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-18T01:09:11.799 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-18T01:09:11.799 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-18T01:09:12.035 INFO:teuthology.orchestra.run.smithi090.stdout:(1/6): ceph-mgr-cephadm-17.2.7-684.g6716c9f3.el 465 kB/s | 123 kB 00:00 2024-03-18T01:09:12.161 INFO:teuthology.orchestra.run.smithi090.stdout:(2/6): python3-babel-2.9.1-2.el9.noarch.rpm 15 MB/s | 5.8 MB 00:00 2024-03-18T01:09:12.292 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-18T01:09:12.293 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-18T01:09:12.293 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2024-03-18T01:09:12.293 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr-diskprediction-local-2:17.2.7-684.g6716c9f3.el9.noarch 2024-03-18T01:09:12.293 INFO:teuthology.orchestra.run.smithi029.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-18T01:09:12.293 INFO:teuthology.orchestra.run.smithi029.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-18T01:09:12.293 INFO:teuthology.orchestra.run.smithi029.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-18T01:09:12.293 INFO:teuthology.orchestra.run.smithi029.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-18T01:09:12.293 INFO:teuthology.orchestra.run.smithi029.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-18T01:09:12.293 INFO:teuthology.orchestra.run.smithi029.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-18T01:09:12.293 INFO:teuthology.orchestra.run.smithi029.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-18T01:09:12.293 INFO:teuthology.orchestra.run.smithi029.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-18T01:09:12.293 INFO:teuthology.orchestra.run.smithi029.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-18T01:09:12.293 INFO:teuthology.orchestra.run.smithi029.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-18T01:09:12.294 INFO:teuthology.orchestra.run.smithi029.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-18T01:09:12.294 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-18T01:09:12.294 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2024-03-18T01:09:12.327 INFO:teuthology.orchestra.run.smithi090.stdout:(3/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 445 kB/s | 248 kB 00:00 2024-03-18T01:09:12.661 INFO:teuthology.orchestra.run.smithi090.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 115 kB/s | 58 kB 00:00 2024-03-18T01:09:12.756 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install ceph-mgr-rook 2024-03-18T01:09:12.761 INFO:teuthology.orchestra.run.smithi090.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 117 kB/s | 51 kB 00:00 2024-03-18T01:09:13.333 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:01:27 ago on Mon 18 Mar 2024 01:07:46 AM UTC. 2024-03-18T01:09:13.488 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2024-03-18T01:09:13.490 INFO:teuthology.orchestra.run.smithi029.stdout:======================================================================================= 2024-03-18T01:09:13.490 INFO:teuthology.orchestra.run.smithi029.stdout: Package Arch Version Repository Size 2024-03-18T01:09:13.490 INFO:teuthology.orchestra.run.smithi029.stdout:======================================================================================= 2024-03-18T01:09:13.490 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2024-03-18T01:09:13.490 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr-rook noarch 2:17.2.7-684.g6716c9f3.el9 ceph-noarch 47 k 2024-03-18T01:09:13.490 INFO:teuthology.orchestra.run.smithi029.stdout:Installing dependencies: 2024-03-18T01:09:13.490 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-18T01:09:13.490 INFO:teuthology.orchestra.run.smithi029.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-18T01:09:13.490 INFO:teuthology.orchestra.run.smithi029.stdout: python3-google-auth noarch 1:2.28.0-1.el9 epel 203 k 2024-03-18T01:09:13.490 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-18T01:09:13.490 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-18T01:09:13.491 INFO:teuthology.orchestra.run.smithi029.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-18T01:09:13.491 INFO:teuthology.orchestra.run.smithi029.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-18T01:09:13.491 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-18T01:09:13.491 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-18T01:09:13.491 INFO:teuthology.orchestra.run.smithi029.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-18T01:09:13.491 INFO:teuthology.orchestra.run.smithi029.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-18T01:09:13.491 INFO:teuthology.orchestra.run.smithi029.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-18T01:09:13.491 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-18T01:09:13.491 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2024-03-18T01:09:13.491 INFO:teuthology.orchestra.run.smithi029.stdout:======================================================================================= 2024-03-18T01:09:13.491 INFO:teuthology.orchestra.run.smithi029.stdout:Install 13 Packages 2024-03-18T01:09:13.491 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-18T01:09:13.492 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 2.0 M 2024-03-18T01:09:13.492 INFO:teuthology.orchestra.run.smithi029.stdout:Installed size: 26 M 2024-03-18T01:09:13.492 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2024-03-18T01:09:13.781 INFO:teuthology.orchestra.run.smithi029.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 138 kB/s | 18 kB 00:00 2024-03-18T01:09:13.823 INFO:teuthology.orchestra.run.smithi029.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 141 kB/s | 24 kB 00:00 2024-03-18T01:09:13.865 INFO:teuthology.orchestra.run.smithi029.stdout:(3/13): ceph-mgr-rook-17.2.7-684.g6716c9f3.el9. 220 kB/s | 47 kB 00:00 2024-03-18T01:09:13.899 INFO:teuthology.orchestra.run.smithi029.stdout:(4/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 1.5 MB/s | 174 kB 00:00 2024-03-18T01:09:13.933 INFO:teuthology.orchestra.run.smithi029.stdout:(5/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.2 MB/s | 132 kB 00:00 2024-03-18T01:09:13.985 INFO:teuthology.orchestra.run.smithi029.stdout:(6/13): python3-pyasn1-modules-0.4.8-6.el9.noar 1.7 MB/s | 211 kB 00:00 2024-03-18T01:09:14.011 INFO:teuthology.orchestra.run.smithi029.stdout:(7/13): python3-requests-oauthlib-1.3.0-12.el9. 384 kB/s | 43 kB 00:00 2024-03-18T01:09:14.278 INFO:teuthology.orchestra.run.smithi090.stdout:(6/6): python3-asyncssh-2.13.2-4.el9.noarch.rpm 244 kB/s | 548 kB 00:02 2024-03-18T01:09:14.280 INFO:teuthology.orchestra.run.smithi090.stdout:-------------------------------------------------------------------------------- 2024-03-18T01:09:14.280 INFO:teuthology.orchestra.run.smithi090.stdout:Total 2.3 MB/s | 6.8 MB 00:02 2024-03-18T01:09:14.295 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction check 2024-03-18T01:09:14.306 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction check succeeded. 2024-03-18T01:09:14.306 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction test 2024-03-18T01:09:14.361 INFO:teuthology.orchestra.run.smithi029.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 40 kB/s | 15 kB 00:00 2024-03-18T01:09:14.387 INFO:teuthology.orchestra.run.smithi029.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 71 kB/s | 32 kB 00:00 2024-03-18T01:09:14.400 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction test succeeded. 2024-03-18T01:09:14.401 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction 2024-03-18T01:09:14.554 INFO:teuthology.orchestra.run.smithi029.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 354 kB/s | 59 kB 00:00 2024-03-18T01:09:14.588 INFO:teuthology.orchestra.run.smithi029.stdout:(11/13): python3-google-auth-2.28.0-1.el9.noarc 351 kB/s | 203 kB 00:00 2024-03-18T01:09:14.699 INFO:teuthology.orchestra.run.smithi090.stdout: Preparing : 1/1 2024-03-18T01:09:14.730 INFO:teuthology.orchestra.run.smithi029.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 512 kB/s | 90 kB 00:00 2024-03-18T01:09:14.877 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-18T01:09:14.905 INFO:teuthology.orchestra.run.smithi029.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 1.9 MB/s | 1.0 MB 00:00 2024-03-18T01:09:14.907 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2024-03-18T01:09:14.908 INFO:teuthology.orchestra.run.smithi029.stdout:Total 1.4 MB/s | 2.0 MB 00:01 2024-03-18T01:09:14.908 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2024-03-18T01:09:14.923 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2024-03-18T01:09:14.924 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2024-03-18T01:09:14.930 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-asyncssh-2.13.2-4.el9.noarch 2/6 2024-03-18T01:09:15.035 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2024-03-18T01:09:15.035 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2024-03-18T01:09:15.438 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-18T01:09:15.526 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-18T01:09:15.572 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2024-03-18T01:09:15.598 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-18T01:09:15.628 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-mgr-cephadm-2:17.2.7-684.g6716c9f3.el9.noarch 6/6 2024-03-18T01:09:15.751 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-18T01:09:15.809 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-18T01:09:15.876 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-18T01:09:15.912 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-18T01:09:15.955 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-18T01:09:16.059 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-18T01:09:16.170 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-google-auth-1:2.28.0-1.el9.noarch 7/13 2024-03-18T01:09:16.233 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-18T01:09:16.796 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.2.7-684.g6716c9f3.el9.noarch 6/6 2024-03-18T01:09:16.796 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-mgr-cephadm-2:17.2.7-684.g6716c9f3.el9.noarch 1/6 2024-03-18T01:09:16.796 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-18T01:09:16.796 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-18T01:09:16.796 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-asyncssh-2.13.2-4.el9.noarch 4/6 2024-03-18T01:09:16.796 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-18T01:09:16.994 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-18T01:09:17.097 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-18T01:09:17.097 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-18T01:09:17.097 INFO:teuthology.orchestra.run.smithi090.stdout:Installed: 2024-03-18T01:09:17.097 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mgr-cephadm-2:17.2.7-684.g6716c9f3.el9.noarch 2024-03-18T01:09:17.097 INFO:teuthology.orchestra.run.smithi090.stdout: python3-asyncssh-2.13.2-4.el9.noarch 2024-03-18T01:09:17.097 INFO:teuthology.orchestra.run.smithi090.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-18T01:09:17.097 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-18T01:09:17.097 INFO:teuthology.orchestra.run.smithi090.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-18T01:09:17.097 INFO:teuthology.orchestra.run.smithi090.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-18T01:09:17.097 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-18T01:09:17.097 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-03-18T01:09:17.102 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-18T01:09:17.136 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-18T01:09:17.185 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-18T01:09:17.208 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-mgr-rook-2:17.2.7-684.g6716c9f3.el9.noarch 13/13 2024-03-18T01:09:17.435 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install ceph-fuse 2024-03-18T01:09:18.000 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:01:50 ago on Mon 18 Mar 2024 01:07:27 AM UTC. 2024-03-18T01:09:18.154 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-03-18T01:09:18.154 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-03-18T01:09:18.154 INFO:teuthology.orchestra.run.smithi090.stdout: Package Arch Version Repository Size 2024-03-18T01:09:18.154 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-03-18T01:09:18.154 INFO:teuthology.orchestra.run.smithi090.stdout:Installing: 2024-03-18T01:09:18.154 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-fuse x86_64 2:17.2.7-684.g6716c9f3.el9 ceph 813 k 2024-03-18T01:09:18.154 INFO:teuthology.orchestra.run.smithi090.stdout:Installing dependencies: 2024-03-18T01:09:18.154 INFO:teuthology.orchestra.run.smithi090.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-18T01:09:18.154 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-18T01:09:18.154 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction Summary 2024-03-18T01:09:18.155 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-03-18T01:09:18.155 INFO:teuthology.orchestra.run.smithi090.stdout:Install 2 Packages 2024-03-18T01:09:18.155 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-18T01:09:18.155 INFO:teuthology.orchestra.run.smithi090.stdout:Total download size: 892 k 2024-03-18T01:09:18.155 INFO:teuthology.orchestra.run.smithi090.stdout:Installed size: 2.6 M 2024-03-18T01:09:18.155 INFO:teuthology.orchestra.run.smithi090.stdout:Downloading Packages: 2024-03-18T01:09:18.253 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-mgr-rook-2:17.2.7-684.g6716c9f3.el9.noarch 13/13 2024-03-18T01:09:18.253 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-mgr-rook-2:17.2.7-684.g6716c9f3.el9.noarch 1/13 2024-03-18T01:09:18.253 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-18T01:09:18.253 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-18T01:09:18.253 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-18T01:09:18.253 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-18T01:09:18.253 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-18T01:09:18.253 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-18T01:09:18.253 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-18T01:09:18.253 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-18T01:09:18.253 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-google-auth-1:2.28.0-1.el9.noarch 10/13 2024-03-18T01:09:18.254 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-18T01:09:18.254 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-18T01:09:18.334 INFO:teuthology.orchestra.run.smithi090.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 441 kB/s | 79 kB 00:00 2024-03-18T01:09:18.535 INFO:teuthology.orchestra.run.smithi090.stdout:(2/2): ceph-fuse-17.2.7-684.g6716c9f3.el9.x86_6 2.1 MB/s | 813 kB 00:00 2024-03-18T01:09:18.536 INFO:teuthology.orchestra.run.smithi090.stdout:-------------------------------------------------------------------------------- 2024-03-18T01:09:18.536 INFO:teuthology.orchestra.run.smithi090.stdout:Total 2.3 MB/s | 892 kB 00:00 2024-03-18T01:09:18.536 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction check 2024-03-18T01:09:18.557 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction check succeeded. 2024-03-18T01:09:18.557 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction test 2024-03-18T01:09:18.632 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction test succeeded. 2024-03-18T01:09:18.632 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction 2024-03-18T01:09:18.632 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-18T01:09:18.632 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-18T01:09:18.632 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2024-03-18T01:09:18.632 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr-rook-2:17.2.7-684.g6716c9f3.el9.noarch 2024-03-18T01:09:18.632 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-18T01:09:18.633 INFO:teuthology.orchestra.run.smithi029.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-18T01:09:18.633 INFO:teuthology.orchestra.run.smithi029.stdout: python3-google-auth-1:2.28.0-1.el9.noarch 2024-03-18T01:09:18.633 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-18T01:09:18.633 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-18T01:09:18.633 INFO:teuthology.orchestra.run.smithi029.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-18T01:09:18.633 INFO:teuthology.orchestra.run.smithi029.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-18T01:09:18.633 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-18T01:09:18.633 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-18T01:09:18.633 INFO:teuthology.orchestra.run.smithi029.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-18T01:09:18.633 INFO:teuthology.orchestra.run.smithi029.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-18T01:09:18.633 INFO:teuthology.orchestra.run.smithi029.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-18T01:09:18.633 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-18T01:09:18.633 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2024-03-18T01:09:18.813 INFO:teuthology.orchestra.run.smithi090.stdout: Preparing : 1/1 2024-03-18T01:09:18.926 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install ceph-mgr-cephadm 2024-03-18T01:09:18.947 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-18T01:09:19.026 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-fuse-2:17.2.7-684.g6716c9f3.el9.x86_64 2/2 2024-03-18T01:09:19.499 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:01:33 ago on Mon 18 Mar 2024 01:07:46 AM UTC. 2024-03-18T01:09:19.655 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2024-03-18T01:09:19.656 INFO:teuthology.orchestra.run.smithi029.stdout:======================================================================================= 2024-03-18T01:09:19.656 INFO:teuthology.orchestra.run.smithi029.stdout: Package Arch Version Repository Size 2024-03-18T01:09:19.656 INFO:teuthology.orchestra.run.smithi029.stdout:======================================================================================= 2024-03-18T01:09:19.656 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2024-03-18T01:09:19.656 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr-cephadm noarch 2:17.2.7-684.g6716c9f3.el9 ceph-noarch 123 k 2024-03-18T01:09:19.656 INFO:teuthology.orchestra.run.smithi029.stdout:Installing dependencies: 2024-03-18T01:09:19.656 INFO:teuthology.orchestra.run.smithi029.stdout: python3-asyncssh noarch 2.13.2-4.el9 epel 548 k 2024-03-18T01:09:19.656 INFO:teuthology.orchestra.run.smithi029.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-18T01:09:19.656 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-18T01:09:19.656 INFO:teuthology.orchestra.run.smithi029.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-18T01:09:19.656 INFO:teuthology.orchestra.run.smithi029.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-18T01:09:19.656 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-18T01:09:19.657 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2024-03-18T01:09:19.657 INFO:teuthology.orchestra.run.smithi029.stdout:======================================================================================= 2024-03-18T01:09:19.657 INFO:teuthology.orchestra.run.smithi029.stdout:Install 6 Packages 2024-03-18T01:09:19.657 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-18T01:09:19.657 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 6.8 M 2024-03-18T01:09:19.657 INFO:teuthology.orchestra.run.smithi029.stdout:Installed size: 33 M 2024-03-18T01:09:19.657 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2024-03-18T01:09:19.858 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-fuse-2:17.2.7-684.g6716c9f3.el9.x86_64 2/2 2024-03-18T01:09:19.859 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-fuse-2:17.2.7-684.g6716c9f3.el9.x86_64 1/2 2024-03-18T01:09:20.181 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-18T01:09:20.181 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-18T01:09:20.181 INFO:teuthology.orchestra.run.smithi090.stdout:Installed: 2024-03-18T01:09:20.181 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-fuse-2:17.2.7-684.g6716c9f3.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-18T01:09:20.182 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-18T01:09:20.182 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-03-18T01:09:20.542 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install librados-devel 2024-03-18T01:09:21.109 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:01:54 ago on Mon 18 Mar 2024 01:07:27 AM UTC. 2024-03-18T01:09:21.259 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-03-18T01:09:21.259 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-03-18T01:09:21.259 INFO:teuthology.orchestra.run.smithi090.stdout: Package Arch Version Repo Size 2024-03-18T01:09:21.260 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-03-18T01:09:21.260 INFO:teuthology.orchestra.run.smithi090.stdout:Installing: 2024-03-18T01:09:21.260 INFO:teuthology.orchestra.run.smithi090.stdout: librados-devel x86_64 2:17.2.7-684.g6716c9f3.el9 ceph 126 k 2024-03-18T01:09:21.260 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-18T01:09:21.260 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction Summary 2024-03-18T01:09:21.260 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-03-18T01:09:21.260 INFO:teuthology.orchestra.run.smithi090.stdout:Install 1 Package 2024-03-18T01:09:21.260 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-18T01:09:21.260 INFO:teuthology.orchestra.run.smithi090.stdout:Total download size: 126 k 2024-03-18T01:09:21.260 INFO:teuthology.orchestra.run.smithi090.stdout:Installed size: 448 k 2024-03-18T01:09:21.260 INFO:teuthology.orchestra.run.smithi090.stdout:Downloading Packages: 2024-03-18T01:09:21.477 INFO:teuthology.orchestra.run.smithi090.stdout:librados-devel-17.2.7-684.g6716c9f3.el9.x86_64. 583 kB/s | 126 kB 00:00 2024-03-18T01:09:21.478 INFO:teuthology.orchestra.run.smithi090.stdout:-------------------------------------------------------------------------------- 2024-03-18T01:09:21.478 INFO:teuthology.orchestra.run.smithi090.stdout:Total 578 kB/s | 126 kB 00:00 2024-03-18T01:09:21.478 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction check 2024-03-18T01:09:21.484 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction check succeeded. 2024-03-18T01:09:21.484 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction test 2024-03-18T01:09:21.538 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction test succeeded. 2024-03-18T01:09:21.538 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction 2024-03-18T01:09:21.639 INFO:teuthology.orchestra.run.smithi029.stdout:(1/6): ceph-mgr-cephadm-17.2.7-684.g6716c9f3.el 522 kB/s | 123 kB 00:00 2024-03-18T01:09:21.676 INFO:teuthology.orchestra.run.smithi090.stdout: Preparing : 1/1 2024-03-18T01:09:21.756 INFO:teuthology.orchestra.run.smithi029.stdout:(2/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 703 kB/s | 248 kB 00:00 2024-03-18T01:09:21.777 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : librados-devel-2:17.2.7-684.g6716c9f3.el9.x86_64 1/1 2024-03-18T01:09:22.131 INFO:teuthology.orchestra.run.smithi029.stdout:(3/6): python3-babel-2.9.1-2.el9.noarch.rpm 7.9 MB/s | 5.8 MB 00:00 2024-03-18T01:09:22.157 INFO:teuthology.orchestra.run.smithi029.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 144 kB/s | 58 kB 00:00 2024-03-18T01:09:22.371 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: librados-devel-2:17.2.7-684.g6716c9f3.el9.x86_64 1/1 2024-03-18T01:09:22.575 INFO:teuthology.orchestra.run.smithi029.stdout:(5/6): python3-asyncssh-2.13.2-4.el9.noarch.rpm 585 kB/s | 548 kB 00:00 2024-03-18T01:09:22.643 INFO:teuthology.orchestra.run.smithi029.stdout:(6/6): python3-typing-extensions-4.4.0-2.el9.no 100 kB/s | 51 kB 00:00 2024-03-18T01:09:22.646 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2024-03-18T01:09:22.646 INFO:teuthology.orchestra.run.smithi029.stdout:Total 2.3 MB/s | 6.8 MB 00:02 2024-03-18T01:09:22.659 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : librados-devel-2:17.2.7-684.g6716c9f3.el9.x86_64 1/1 2024-03-18T01:09:22.659 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-18T01:09:22.659 INFO:teuthology.orchestra.run.smithi090.stdout:Installed: 2024-03-18T01:09:22.659 INFO:teuthology.orchestra.run.smithi090.stdout: librados-devel-2:17.2.7-684.g6716c9f3.el9.x86_64 2024-03-18T01:09:22.659 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-18T01:09:22.659 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-03-18T01:09:22.660 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2024-03-18T01:09:22.671 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2024-03-18T01:09:22.671 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2024-03-18T01:09:22.768 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2024-03-18T01:09:22.768 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2024-03-18T01:09:22.973 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install libcephfs2 2024-03-18T01:09:23.073 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2024-03-18T01:09:23.272 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-18T01:09:23.338 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-asyncssh-2.13.2-4.el9.noarch 2/6 2024-03-18T01:09:23.548 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:01:56 ago on Mon 18 Mar 2024 01:07:27 AM UTC. 2024-03-18T01:09:23.649 INFO:teuthology.orchestra.run.smithi090.stdout:Package libcephfs2-2:17.2.7-684.g6716c9f3.el9.x86_64 is already installed. 2024-03-18T01:09:23.702 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-03-18T01:09:23.703 INFO:teuthology.orchestra.run.smithi090.stdout:Nothing to do. 2024-03-18T01:09:23.703 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-03-18T01:09:23.778 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install libcephfs-devel 2024-03-18T01:09:23.895 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-18T01:09:23.996 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-18T01:09:24.059 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-18T01:09:24.096 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-mgr-cephadm-2:17.2.7-684.g6716c9f3.el9.noarch 6/6 2024-03-18T01:09:24.339 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:01:57 ago on Mon 18 Mar 2024 01:07:27 AM UTC. 2024-03-18T01:09:24.486 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-03-18T01:09:24.487 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-03-18T01:09:24.487 INFO:teuthology.orchestra.run.smithi090.stdout: Package Arch Version Repo Size 2024-03-18T01:09:24.487 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-03-18T01:09:24.487 INFO:teuthology.orchestra.run.smithi090.stdout:Installing: 2024-03-18T01:09:24.487 INFO:teuthology.orchestra.run.smithi090.stdout: libcephfs-devel x86_64 2:17.2.7-684.g6716c9f3.el9 ceph 25 k 2024-03-18T01:09:24.488 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-18T01:09:24.488 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction Summary 2024-03-18T01:09:24.488 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-03-18T01:09:24.488 INFO:teuthology.orchestra.run.smithi090.stdout:Install 1 Package 2024-03-18T01:09:24.488 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-18T01:09:24.488 INFO:teuthology.orchestra.run.smithi090.stdout:Total download size: 25 k 2024-03-18T01:09:24.488 INFO:teuthology.orchestra.run.smithi090.stdout:Installed size: 110 k 2024-03-18T01:09:24.488 INFO:teuthology.orchestra.run.smithi090.stdout:Downloading Packages: 2024-03-18T01:09:24.662 INFO:teuthology.orchestra.run.smithi090.stdout:libcephfs-devel-17.2.7-684.g6716c9f3.el9.x86_64 144 kB/s | 25 kB 00:00 2024-03-18T01:09:24.662 INFO:teuthology.orchestra.run.smithi090.stdout:-------------------------------------------------------------------------------- 2024-03-18T01:09:24.663 INFO:teuthology.orchestra.run.smithi090.stdout:Total 143 kB/s | 25 kB 00:00 2024-03-18T01:09:24.663 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction check 2024-03-18T01:09:24.666 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction check succeeded. 2024-03-18T01:09:24.666 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction test 2024-03-18T01:09:24.667 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction test succeeded. 2024-03-18T01:09:24.668 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction 2024-03-18T01:09:24.747 INFO:teuthology.orchestra.run.smithi090.stdout: Preparing : 1/1 2024-03-18T01:09:24.860 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : libcephfs-devel-2:17.2.7-684.g6716c9f3.el9.x86_64 1/1 2024-03-18T01:09:25.212 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.2.7-684.g6716c9f3.el9.noarch 6/6 2024-03-18T01:09:25.212 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-mgr-cephadm-2:17.2.7-684.g6716c9f3.el9.noarch 1/6 2024-03-18T01:09:25.212 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-18T01:09:25.212 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-18T01:09:25.212 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-asyncssh-2.13.2-4.el9.noarch 4/6 2024-03-18T01:09:25.213 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-18T01:09:25.355 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: libcephfs-devel-2:17.2.7-684.g6716c9f3.el9.x86_64 1/1 2024-03-18T01:09:25.539 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-18T01:09:25.539 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-18T01:09:25.539 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2024-03-18T01:09:25.539 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr-cephadm-2:17.2.7-684.g6716c9f3.el9.noarch 2024-03-18T01:09:25.539 INFO:teuthology.orchestra.run.smithi029.stdout: python3-asyncssh-2.13.2-4.el9.noarch 2024-03-18T01:09:25.539 INFO:teuthology.orchestra.run.smithi029.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-18T01:09:25.539 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-18T01:09:25.539 INFO:teuthology.orchestra.run.smithi029.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-18T01:09:25.539 INFO:teuthology.orchestra.run.smithi029.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-18T01:09:25.539 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-18T01:09:25.540 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2024-03-18T01:09:25.615 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : libcephfs-devel-2:17.2.7-684.g6716c9f3.el9.x86_64 1/1 2024-03-18T01:09:25.615 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-18T01:09:25.615 INFO:teuthology.orchestra.run.smithi090.stdout:Installed: 2024-03-18T01:09:25.615 INFO:teuthology.orchestra.run.smithi090.stdout: libcephfs-devel-2:17.2.7-684.g6716c9f3.el9.x86_64 2024-03-18T01:09:25.615 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-18T01:09:25.615 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-03-18T01:09:25.839 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install librados2 2024-03-18T01:09:25.855 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install ceph-fuse 2024-03-18T01:09:26.398 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:01:59 ago on Mon 18 Mar 2024 01:07:27 AM UTC. 2024-03-18T01:09:26.441 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:01:40 ago on Mon 18 Mar 2024 01:07:46 AM UTC. 2024-03-18T01:09:26.494 INFO:teuthology.orchestra.run.smithi090.stdout:Package librados2-2:17.2.7-684.g6716c9f3.el9.x86_64 is already installed. 2024-03-18T01:09:26.546 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-03-18T01:09:26.547 INFO:teuthology.orchestra.run.smithi090.stdout:Nothing to do. 2024-03-18T01:09:26.547 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-03-18T01:09:26.597 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2024-03-18T01:09:26.597 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-03-18T01:09:26.598 INFO:teuthology.orchestra.run.smithi029.stdout: Package Arch Version Repository Size 2024-03-18T01:09:26.598 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-03-18T01:09:26.598 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2024-03-18T01:09:26.598 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-fuse x86_64 2:17.2.7-684.g6716c9f3.el9 ceph 813 k 2024-03-18T01:09:26.598 INFO:teuthology.orchestra.run.smithi029.stdout:Installing dependencies: 2024-03-18T01:09:26.598 INFO:teuthology.orchestra.run.smithi029.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-18T01:09:26.598 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-18T01:09:26.598 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2024-03-18T01:09:26.598 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-03-18T01:09:26.598 INFO:teuthology.orchestra.run.smithi029.stdout:Install 2 Packages 2024-03-18T01:09:26.598 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-18T01:09:26.599 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 892 k 2024-03-18T01:09:26.599 INFO:teuthology.orchestra.run.smithi029.stdout:Installed size: 2.6 M 2024-03-18T01:09:26.599 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2024-03-18T01:09:26.650 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install librbd1 2024-03-18T01:09:26.733 INFO:teuthology.orchestra.run.smithi029.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 588 kB/s | 79 kB 00:00 2024-03-18T01:09:27.100 INFO:teuthology.orchestra.run.smithi029.stdout:(2/2): ceph-fuse-17.2.7-684.g6716c9f3.el9.x86_6 1.6 MB/s | 813 kB 00:00 2024-03-18T01:09:27.101 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2024-03-18T01:09:27.102 INFO:teuthology.orchestra.run.smithi029.stdout:Total 1.7 MB/s | 892 kB 00:00 2024-03-18T01:09:27.102 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2024-03-18T01:09:27.123 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2024-03-18T01:09:27.123 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2024-03-18T01:09:27.201 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2024-03-18T01:09:27.202 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2024-03-18T01:09:27.208 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:02:00 ago on Mon 18 Mar 2024 01:07:27 AM UTC. 2024-03-18T01:09:27.305 INFO:teuthology.orchestra.run.smithi090.stdout:Package librbd1-2:17.2.7-684.g6716c9f3.el9.x86_64 is already installed. 2024-03-18T01:09:27.357 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-03-18T01:09:27.358 INFO:teuthology.orchestra.run.smithi090.stdout:Nothing to do. 2024-03-18T01:09:27.358 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-03-18T01:09:27.389 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2024-03-18T01:09:27.426 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install python3-rados 2024-03-18T01:09:27.525 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-18T01:09:27.606 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-fuse-2:17.2.7-684.g6716c9f3.el9.x86_64 2/2 2024-03-18T01:09:27.987 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:02:00 ago on Mon 18 Mar 2024 01:07:27 AM UTC. 2024-03-18T01:09:28.083 INFO:teuthology.orchestra.run.smithi090.stdout:Package python3-rados-2:17.2.7-684.g6716c9f3.el9.x86_64 is already installed. 2024-03-18T01:09:28.135 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-03-18T01:09:28.136 INFO:teuthology.orchestra.run.smithi090.stdout:Nothing to do. 2024-03-18T01:09:28.137 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-03-18T01:09:28.236 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install python3-rgw 2024-03-18T01:09:28.394 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-fuse-2:17.2.7-684.g6716c9f3.el9.x86_64 2/2 2024-03-18T01:09:28.394 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-fuse-2:17.2.7-684.g6716c9f3.el9.x86_64 1/2 2024-03-18T01:09:28.726 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-18T01:09:28.726 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-18T01:09:28.726 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2024-03-18T01:09:28.726 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-fuse-2:17.2.7-684.g6716c9f3.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-18T01:09:28.727 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-18T01:09:28.727 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2024-03-18T01:09:28.800 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:02:01 ago on Mon 18 Mar 2024 01:07:27 AM UTC. 2024-03-18T01:09:28.898 INFO:teuthology.orchestra.run.smithi090.stdout:Package python3-rgw-2:17.2.7-684.g6716c9f3.el9.x86_64 is already installed. 2024-03-18T01:09:28.955 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-03-18T01:09:28.955 INFO:teuthology.orchestra.run.smithi090.stdout:Nothing to do. 2024-03-18T01:09:28.955 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-03-18T01:09:29.025 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install python3-cephfs 2024-03-18T01:09:29.040 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install librados-devel 2024-03-18T01:09:29.591 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:02:02 ago on Mon 18 Mar 2024 01:07:27 AM UTC. 2024-03-18T01:09:29.622 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:01:43 ago on Mon 18 Mar 2024 01:07:46 AM UTC. 2024-03-18T01:09:29.688 INFO:teuthology.orchestra.run.smithi090.stdout:Package python3-cephfs-2:17.2.7-684.g6716c9f3.el9.x86_64 is already installed. 2024-03-18T01:09:29.740 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-03-18T01:09:29.741 INFO:teuthology.orchestra.run.smithi090.stdout:Nothing to do. 2024-03-18T01:09:29.741 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-03-18T01:09:29.777 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2024-03-18T01:09:29.778 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-03-18T01:09:29.778 INFO:teuthology.orchestra.run.smithi029.stdout: Package Arch Version Repo Size 2024-03-18T01:09:29.778 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-03-18T01:09:29.778 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2024-03-18T01:09:29.778 INFO:teuthology.orchestra.run.smithi029.stdout: librados-devel x86_64 2:17.2.7-684.g6716c9f3.el9 ceph 126 k 2024-03-18T01:09:29.778 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-18T01:09:29.778 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2024-03-18T01:09:29.778 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-03-18T01:09:29.778 INFO:teuthology.orchestra.run.smithi029.stdout:Install 1 Package 2024-03-18T01:09:29.778 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-18T01:09:29.779 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 126 k 2024-03-18T01:09:29.779 INFO:teuthology.orchestra.run.smithi029.stdout:Installed size: 448 k 2024-03-18T01:09:29.779 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2024-03-18T01:09:29.811 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install python3-rbd 2024-03-18T01:09:30.104 INFO:teuthology.orchestra.run.smithi029.stdout:librados-devel-17.2.7-684.g6716c9f3.el9.x86_64. 388 kB/s | 126 kB 00:00 2024-03-18T01:09:30.105 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2024-03-18T01:09:30.105 INFO:teuthology.orchestra.run.smithi029.stdout:Total 386 kB/s | 126 kB 00:00 2024-03-18T01:09:30.105 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2024-03-18T01:09:30.111 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2024-03-18T01:09:30.111 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2024-03-18T01:09:30.170 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2024-03-18T01:09:30.170 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2024-03-18T01:09:30.317 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2024-03-18T01:09:30.371 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:02:03 ago on Mon 18 Mar 2024 01:07:27 AM UTC. 2024-03-18T01:09:30.425 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : librados-devel-2:17.2.7-684.g6716c9f3.el9.x86_64 1/1 2024-03-18T01:09:30.469 INFO:teuthology.orchestra.run.smithi090.stdout:Package python3-rbd-2:17.2.7-684.g6716c9f3.el9.x86_64 is already installed. 2024-03-18T01:09:30.522 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-03-18T01:09:30.523 INFO:teuthology.orchestra.run.smithi090.stdout:Nothing to do. 2024-03-18T01:09:30.523 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-03-18T01:09:30.594 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install rbd-fuse 2024-03-18T01:09:30.958 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: librados-devel-2:17.2.7-684.g6716c9f3.el9.x86_64 1/1 2024-03-18T01:09:31.156 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:02:04 ago on Mon 18 Mar 2024 01:07:27 AM UTC. 2024-03-18T01:09:31.263 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : librados-devel-2:17.2.7-684.g6716c9f3.el9.x86_64 1/1 2024-03-18T01:09:31.263 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-18T01:09:31.264 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2024-03-18T01:09:31.264 INFO:teuthology.orchestra.run.smithi029.stdout: librados-devel-2:17.2.7-684.g6716c9f3.el9.x86_64 2024-03-18T01:09:31.264 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-18T01:09:31.264 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2024-03-18T01:09:31.306 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-03-18T01:09:31.306 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-03-18T01:09:31.306 INFO:teuthology.orchestra.run.smithi090.stdout: Package Architecture Version Repository Size 2024-03-18T01:09:31.306 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-03-18T01:09:31.306 INFO:teuthology.orchestra.run.smithi090.stdout:Installing: 2024-03-18T01:09:31.306 INFO:teuthology.orchestra.run.smithi090.stdout: rbd-fuse x86_64 2:17.2.7-684.g6716c9f3.el9 ceph 82 k 2024-03-18T01:09:31.307 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-18T01:09:31.307 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction Summary 2024-03-18T01:09:31.307 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-03-18T01:09:31.307 INFO:teuthology.orchestra.run.smithi090.stdout:Install 1 Package 2024-03-18T01:09:31.307 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-18T01:09:31.307 INFO:teuthology.orchestra.run.smithi090.stdout:Total download size: 82 k 2024-03-18T01:09:31.307 INFO:teuthology.orchestra.run.smithi090.stdout:Installed size: 222 k 2024-03-18T01:09:31.308 INFO:teuthology.orchestra.run.smithi090.stdout:Downloading Packages: 2024-03-18T01:09:31.615 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install libcephfs2 2024-03-18T01:09:31.634 INFO:teuthology.orchestra.run.smithi090.stdout:rbd-fuse-17.2.7-684.g6716c9f3.el9.x86_64.rpm 250 kB/s | 82 kB 00:00 2024-03-18T01:09:31.635 INFO:teuthology.orchestra.run.smithi090.stdout:-------------------------------------------------------------------------------- 2024-03-18T01:09:31.635 INFO:teuthology.orchestra.run.smithi090.stdout:Total 249 kB/s | 82 kB 00:00 2024-03-18T01:09:31.635 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction check 2024-03-18T01:09:31.642 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction check succeeded. 2024-03-18T01:09:31.642 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction test 2024-03-18T01:09:31.696 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction test succeeded. 2024-03-18T01:09:31.696 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction 2024-03-18T01:09:31.827 INFO:teuthology.orchestra.run.smithi090.stdout: Preparing : 1/1 2024-03-18T01:09:31.924 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : rbd-fuse-2:17.2.7-684.g6716c9f3.el9.x86_64 1/1 2024-03-18T01:09:32.193 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:01:46 ago on Mon 18 Mar 2024 01:07:46 AM UTC. 2024-03-18T01:09:32.292 INFO:teuthology.orchestra.run.smithi029.stdout:Package libcephfs2-2:17.2.7-684.g6716c9f3.el9.x86_64 is already installed. 2024-03-18T01:09:32.348 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2024-03-18T01:09:32.349 INFO:teuthology.orchestra.run.smithi029.stdout:Nothing to do. 2024-03-18T01:09:32.350 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2024-03-18T01:09:32.413 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: rbd-fuse-2:17.2.7-684.g6716c9f3.el9.x86_64 1/1 2024-03-18T01:09:32.431 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install libcephfs-devel 2024-03-18T01:09:32.699 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : rbd-fuse-2:17.2.7-684.g6716c9f3.el9.x86_64 1/1 2024-03-18T01:09:32.699 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-18T01:09:32.699 INFO:teuthology.orchestra.run.smithi090.stdout:Installed: 2024-03-18T01:09:32.700 INFO:teuthology.orchestra.run.smithi090.stdout: rbd-fuse-2:17.2.7-684.g6716c9f3.el9.x86_64 2024-03-18T01:09:32.700 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-18T01:09:32.700 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-03-18T01:09:33.005 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:01:47 ago on Mon 18 Mar 2024 01:07:46 AM UTC. 2024-03-18T01:09:33.020 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install rbd-mirror 2024-03-18T01:09:33.161 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2024-03-18T01:09:33.162 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-03-18T01:09:33.162 INFO:teuthology.orchestra.run.smithi029.stdout: Package Arch Version Repo Size 2024-03-18T01:09:33.162 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-03-18T01:09:33.162 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2024-03-18T01:09:33.163 INFO:teuthology.orchestra.run.smithi029.stdout: libcephfs-devel x86_64 2:17.2.7-684.g6716c9f3.el9 ceph 25 k 2024-03-18T01:09:33.163 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-18T01:09:33.163 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2024-03-18T01:09:33.163 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-03-18T01:09:33.163 INFO:teuthology.orchestra.run.smithi029.stdout:Install 1 Package 2024-03-18T01:09:33.163 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-18T01:09:33.163 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 25 k 2024-03-18T01:09:33.163 INFO:teuthology.orchestra.run.smithi029.stdout:Installed size: 110 k 2024-03-18T01:09:33.163 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2024-03-18T01:09:33.532 INFO:teuthology.orchestra.run.smithi029.stdout:libcephfs-devel-17.2.7-684.g6716c9f3.el9.x86_64 68 kB/s | 25 kB 00:00 2024-03-18T01:09:33.532 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2024-03-18T01:09:33.533 INFO:teuthology.orchestra.run.smithi029.stdout:Total 67 kB/s | 25 kB 00:00 2024-03-18T01:09:33.533 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2024-03-18T01:09:33.536 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2024-03-18T01:09:33.537 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2024-03-18T01:09:33.538 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2024-03-18T01:09:33.538 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2024-03-18T01:09:33.595 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:02:06 ago on Mon 18 Mar 2024 01:07:27 AM UTC. 2024-03-18T01:09:33.627 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2024-03-18T01:09:33.752 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-03-18T01:09:33.753 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-03-18T01:09:33.753 INFO:teuthology.orchestra.run.smithi090.stdout: Package Arch Version Repo Size 2024-03-18T01:09:33.753 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-03-18T01:09:33.753 INFO:teuthology.orchestra.run.smithi090.stdout:Installing: 2024-03-18T01:09:33.753 INFO:teuthology.orchestra.run.smithi090.stdout: rbd-mirror x86_64 2:17.2.7-684.g6716c9f3.el9 ceph 2.9 M 2024-03-18T01:09:33.753 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-18T01:09:33.753 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction Summary 2024-03-18T01:09:33.753 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-03-18T01:09:33.753 INFO:teuthology.orchestra.run.smithi090.stdout:Install 1 Package 2024-03-18T01:09:33.753 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-18T01:09:33.753 INFO:teuthology.orchestra.run.smithi090.stdout:Total download size: 2.9 M 2024-03-18T01:09:33.754 INFO:teuthology.orchestra.run.smithi090.stdout:Installed size: 12 M 2024-03-18T01:09:33.754 INFO:teuthology.orchestra.run.smithi090.stdout:Downloading Packages: 2024-03-18T01:09:33.767 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : libcephfs-devel-2:17.2.7-684.g6716c9f3.el9.x86_64 1/1 2024-03-18T01:09:34.192 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: libcephfs-devel-2:17.2.7-684.g6716c9f3.el9.x86_64 1/1 2024-03-18T01:09:34.460 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libcephfs-devel-2:17.2.7-684.g6716c9f3.el9.x86_64 1/1 2024-03-18T01:09:34.460 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-18T01:09:34.461 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2024-03-18T01:09:34.461 INFO:teuthology.orchestra.run.smithi029.stdout: libcephfs-devel-2:17.2.7-684.g6716c9f3.el9.x86_64 2024-03-18T01:09:34.461 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-18T01:09:34.461 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2024-03-18T01:09:34.684 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install librados2 2024-03-18T01:09:35.160 INFO:teuthology.orchestra.run.smithi090.stdout:rbd-mirror-17.2.7-684.g6716c9f3.el9.x86_64.rpm 2.1 MB/s | 2.9 MB 00:01 2024-03-18T01:09:35.160 INFO:teuthology.orchestra.run.smithi090.stdout:-------------------------------------------------------------------------------- 2024-03-18T01:09:35.161 INFO:teuthology.orchestra.run.smithi090.stdout:Total 2.1 MB/s | 2.9 MB 00:01 2024-03-18T01:09:35.161 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction check 2024-03-18T01:09:35.171 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction check succeeded. 2024-03-18T01:09:35.171 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction test 2024-03-18T01:09:35.246 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction test succeeded. 2024-03-18T01:09:35.247 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction 2024-03-18T01:09:35.258 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:01:49 ago on Mon 18 Mar 2024 01:07:46 AM UTC. 2024-03-18T01:09:35.358 INFO:teuthology.orchestra.run.smithi029.stdout:Package librados2-2:17.2.7-684.g6716c9f3.el9.x86_64 is already installed. 2024-03-18T01:09:35.414 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2024-03-18T01:09:35.415 INFO:teuthology.orchestra.run.smithi029.stdout:Nothing to do. 2024-03-18T01:09:35.415 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2024-03-18T01:09:35.502 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install librbd1 2024-03-18T01:09:35.650 INFO:teuthology.orchestra.run.smithi090.stdout: Preparing : 1/1 2024-03-18T01:09:35.702 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : rbd-mirror-2:17.2.7-684.g6716c9f3.el9.x86_64 1/1 2024-03-18T01:09:35.725 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: rbd-mirror-2:17.2.7-684.g6716c9f3.el9.x86_64 1/1 2024-03-18T01:09:35.725 INFO:teuthology.orchestra.run.smithi090.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-18T01:09:35.725 INFO:teuthology.orchestra.run.smithi090.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-18T01:09:35.726 INFO:teuthology.orchestra.run.smithi090.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-18T01:09:35.726 INFO:teuthology.orchestra.run.smithi090.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-18T01:09:35.726 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-18T01:09:36.072 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:01:50 ago on Mon 18 Mar 2024 01:07:46 AM UTC. 2024-03-18T01:09:36.172 INFO:teuthology.orchestra.run.smithi029.stdout:Package librbd1-2:17.2.7-684.g6716c9f3.el9.x86_64 is already installed. 2024-03-18T01:09:36.228 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2024-03-18T01:09:36.229 INFO:teuthology.orchestra.run.smithi029.stdout:Nothing to do. 2024-03-18T01:09:36.229 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2024-03-18T01:09:36.306 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install python3-rados 2024-03-18T01:09:36.886 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:01:50 ago on Mon 18 Mar 2024 01:07:46 AM UTC. 2024-03-18T01:09:36.954 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : rbd-mirror-2:17.2.7-684.g6716c9f3.el9.x86_64 1/1 2024-03-18T01:09:36.954 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-18T01:09:36.955 INFO:teuthology.orchestra.run.smithi090.stdout:Installed: 2024-03-18T01:09:36.955 INFO:teuthology.orchestra.run.smithi090.stdout: rbd-mirror-2:17.2.7-684.g6716c9f3.el9.x86_64 2024-03-18T01:09:36.955 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-18T01:09:36.955 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-03-18T01:09:36.986 INFO:teuthology.orchestra.run.smithi029.stdout:Package python3-rados-2:17.2.7-684.g6716c9f3.el9.x86_64 is already installed. 2024-03-18T01:09:37.041 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2024-03-18T01:09:37.042 INFO:teuthology.orchestra.run.smithi029.stdout:Nothing to do. 2024-03-18T01:09:37.042 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2024-03-18T01:09:37.123 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install python3-rgw 2024-03-18T01:09:37.282 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install rbd-nbd 2024-03-18T01:09:37.700 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:01:51 ago on Mon 18 Mar 2024 01:07:46 AM UTC. 2024-03-18T01:09:37.800 INFO:teuthology.orchestra.run.smithi029.stdout:Package python3-rgw-2:17.2.7-684.g6716c9f3.el9.x86_64 is already installed. 2024-03-18T01:09:37.855 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:02:10 ago on Mon 18 Mar 2024 01:07:27 AM UTC. 2024-03-18T01:09:37.855 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2024-03-18T01:09:37.856 INFO:teuthology.orchestra.run.smithi029.stdout:Nothing to do. 2024-03-18T01:09:37.857 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2024-03-18T01:09:37.932 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install python3-cephfs 2024-03-18T01:09:38.004 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-03-18T01:09:38.005 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-03-18T01:09:38.005 INFO:teuthology.orchestra.run.smithi090.stdout: Package Architecture Version Repository Size 2024-03-18T01:09:38.005 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-03-18T01:09:38.005 INFO:teuthology.orchestra.run.smithi090.stdout:Installing: 2024-03-18T01:09:38.005 INFO:teuthology.orchestra.run.smithi090.stdout: rbd-nbd x86_64 2:17.2.7-684.g6716c9f3.el9 ceph 165 k 2024-03-18T01:09:38.005 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-18T01:09:38.005 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction Summary 2024-03-18T01:09:38.006 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-03-18T01:09:38.006 INFO:teuthology.orchestra.run.smithi090.stdout:Install 1 Package 2024-03-18T01:09:38.006 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-18T01:09:38.006 INFO:teuthology.orchestra.run.smithi090.stdout:Total download size: 165 k 2024-03-18T01:09:38.006 INFO:teuthology.orchestra.run.smithi090.stdout:Installed size: 477 k 2024-03-18T01:09:38.006 INFO:teuthology.orchestra.run.smithi090.stdout:Downloading Packages: 2024-03-18T01:09:38.267 INFO:teuthology.orchestra.run.smithi090.stdout:rbd-nbd-17.2.7-684.g6716c9f3.el9.x86_64.rpm 632 kB/s | 165 kB 00:00 2024-03-18T01:09:38.268 INFO:teuthology.orchestra.run.smithi090.stdout:-------------------------------------------------------------------------------- 2024-03-18T01:09:38.268 INFO:teuthology.orchestra.run.smithi090.stdout:Total 628 kB/s | 165 kB 00:00 2024-03-18T01:09:38.268 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction check 2024-03-18T01:09:38.276 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction check succeeded. 2024-03-18T01:09:38.276 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction test 2024-03-18T01:09:38.330 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction test succeeded. 2024-03-18T01:09:38.330 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction 2024-03-18T01:09:38.471 INFO:teuthology.orchestra.run.smithi090.stdout: Preparing : 1/1 2024-03-18T01:09:38.508 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:01:52 ago on Mon 18 Mar 2024 01:07:46 AM UTC. 2024-03-18T01:09:38.584 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : rbd-nbd-2:17.2.7-684.g6716c9f3.el9.x86_64 1/1 2024-03-18T01:09:38.610 INFO:teuthology.orchestra.run.smithi029.stdout:Package python3-cephfs-2:17.2.7-684.g6716c9f3.el9.x86_64 is already installed. 2024-03-18T01:09:38.663 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2024-03-18T01:09:38.664 INFO:teuthology.orchestra.run.smithi029.stdout:Nothing to do. 2024-03-18T01:09:38.664 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2024-03-18T01:09:38.743 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install python3-rbd 2024-03-18T01:09:39.118 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: rbd-nbd-2:17.2.7-684.g6716c9f3.el9.x86_64 1/1 2024-03-18T01:09:39.315 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:01:53 ago on Mon 18 Mar 2024 01:07:46 AM UTC. 2024-03-18T01:09:39.403 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : rbd-nbd-2:17.2.7-684.g6716c9f3.el9.x86_64 1/1 2024-03-18T01:09:39.404 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-18T01:09:39.404 INFO:teuthology.orchestra.run.smithi090.stdout:Installed: 2024-03-18T01:09:39.404 INFO:teuthology.orchestra.run.smithi090.stdout: rbd-nbd-2:17.2.7-684.g6716c9f3.el9.x86_64 2024-03-18T01:09:39.404 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-18T01:09:39.404 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-03-18T01:09:39.414 INFO:teuthology.orchestra.run.smithi029.stdout:Package python3-rbd-2:17.2.7-684.g6716c9f3.el9.x86_64 is already installed. 2024-03-18T01:09:39.471 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2024-03-18T01:09:39.472 INFO:teuthology.orchestra.run.smithi029.stdout:Nothing to do. 2024-03-18T01:09:39.472 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2024-03-18T01:09:39.547 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install rbd-fuse 2024-03-18T01:09:39.741 DEBUG:teuthology.parallel:result is None 2024-03-18T01:09:40.119 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:01:54 ago on Mon 18 Mar 2024 01:07:46 AM UTC. 2024-03-18T01:09:40.273 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2024-03-18T01:09:40.274 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-03-18T01:09:40.274 INFO:teuthology.orchestra.run.smithi029.stdout: Package Architecture Version Repository Size 2024-03-18T01:09:40.274 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-03-18T01:09:40.274 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2024-03-18T01:09:40.274 INFO:teuthology.orchestra.run.smithi029.stdout: rbd-fuse x86_64 2:17.2.7-684.g6716c9f3.el9 ceph 82 k 2024-03-18T01:09:40.275 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-18T01:09:40.275 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2024-03-18T01:09:40.275 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-03-18T01:09:40.275 INFO:teuthology.orchestra.run.smithi029.stdout:Install 1 Package 2024-03-18T01:09:40.275 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-18T01:09:40.275 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 82 k 2024-03-18T01:09:40.275 INFO:teuthology.orchestra.run.smithi029.stdout:Installed size: 222 k 2024-03-18T01:09:40.275 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2024-03-18T01:09:40.824 INFO:teuthology.orchestra.run.smithi029.stdout:rbd-fuse-17.2.7-684.g6716c9f3.el9.x86_64.rpm 149 kB/s | 82 kB 00:00 2024-03-18T01:09:40.825 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2024-03-18T01:09:40.825 INFO:teuthology.orchestra.run.smithi029.stdout:Total 149 kB/s | 82 kB 00:00 2024-03-18T01:09:40.826 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2024-03-18T01:09:40.832 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2024-03-18T01:09:40.833 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2024-03-18T01:09:40.890 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2024-03-18T01:09:40.891 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2024-03-18T01:09:41.026 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2024-03-18T01:09:41.141 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : rbd-fuse-2:17.2.7-684.g6716c9f3.el9.x86_64 1/1 2024-03-18T01:09:41.628 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: rbd-fuse-2:17.2.7-684.g6716c9f3.el9.x86_64 1/1 2024-03-18T01:09:41.916 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : rbd-fuse-2:17.2.7-684.g6716c9f3.el9.x86_64 1/1 2024-03-18T01:09:41.917 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-18T01:09:41.917 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2024-03-18T01:09:41.917 INFO:teuthology.orchestra.run.smithi029.stdout: rbd-fuse-2:17.2.7-684.g6716c9f3.el9.x86_64 2024-03-18T01:09:41.917 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-18T01:09:41.917 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2024-03-18T01:09:42.216 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install rbd-mirror 2024-03-18T01:09:42.796 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:01:56 ago on Mon 18 Mar 2024 01:07:46 AM UTC. 2024-03-18T01:09:42.956 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2024-03-18T01:09:42.957 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-03-18T01:09:42.958 INFO:teuthology.orchestra.run.smithi029.stdout: Package Arch Version Repo Size 2024-03-18T01:09:42.958 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-03-18T01:09:42.958 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2024-03-18T01:09:42.958 INFO:teuthology.orchestra.run.smithi029.stdout: rbd-mirror x86_64 2:17.2.7-684.g6716c9f3.el9 ceph 2.9 M 2024-03-18T01:09:42.958 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-18T01:09:42.958 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2024-03-18T01:09:42.958 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-03-18T01:09:42.958 INFO:teuthology.orchestra.run.smithi029.stdout:Install 1 Package 2024-03-18T01:09:42.958 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-18T01:09:42.958 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 2.9 M 2024-03-18T01:09:42.958 INFO:teuthology.orchestra.run.smithi029.stdout:Installed size: 12 M 2024-03-18T01:09:42.958 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2024-03-18T01:09:43.342 INFO:teuthology.orchestra.run.smithi029.stdout:rbd-mirror-17.2.7-684.g6716c9f3.el9.x86_64.rpm 7.7 MB/s | 2.9 MB 00:00 2024-03-18T01:09:43.343 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2024-03-18T01:09:43.343 INFO:teuthology.orchestra.run.smithi029.stdout:Total 7.6 MB/s | 2.9 MB 00:00 2024-03-18T01:09:43.343 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2024-03-18T01:09:43.353 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2024-03-18T01:09:43.353 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2024-03-18T01:09:43.428 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2024-03-18T01:09:43.428 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2024-03-18T01:09:43.851 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2024-03-18T01:09:43.919 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : rbd-mirror-2:17.2.7-684.g6716c9f3.el9.x86_64 1/1 2024-03-18T01:09:43.942 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: rbd-mirror-2:17.2.7-684.g6716c9f3.el9.x86_64 1/1 2024-03-18T01:09:43.943 INFO:teuthology.orchestra.run.smithi029.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-18T01:09:43.943 INFO:teuthology.orchestra.run.smithi029.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-18T01:09:43.943 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-18T01:09:43.943 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-18T01:09:43.943 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-18T01:09:45.215 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : rbd-mirror-2:17.2.7-684.g6716c9f3.el9.x86_64 1/1 2024-03-18T01:09:45.215 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-18T01:09:45.216 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2024-03-18T01:09:45.216 INFO:teuthology.orchestra.run.smithi029.stdout: rbd-mirror-2:17.2.7-684.g6716c9f3.el9.x86_64 2024-03-18T01:09:45.216 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-18T01:09:45.216 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2024-03-18T01:09:45.495 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install rbd-nbd 2024-03-18T01:09:46.073 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:02:00 ago on Mon 18 Mar 2024 01:07:46 AM UTC. 2024-03-18T01:09:46.228 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2024-03-18T01:09:46.229 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-03-18T01:09:46.229 INFO:teuthology.orchestra.run.smithi029.stdout: Package Architecture Version Repository Size 2024-03-18T01:09:46.229 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-03-18T01:09:46.229 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2024-03-18T01:09:46.229 INFO:teuthology.orchestra.run.smithi029.stdout: rbd-nbd x86_64 2:17.2.7-684.g6716c9f3.el9 ceph 165 k 2024-03-18T01:09:46.229 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-18T01:09:46.229 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2024-03-18T01:09:46.229 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-03-18T01:09:46.229 INFO:teuthology.orchestra.run.smithi029.stdout:Install 1 Package 2024-03-18T01:09:46.229 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-18T01:09:46.230 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 165 k 2024-03-18T01:09:46.230 INFO:teuthology.orchestra.run.smithi029.stdout:Installed size: 477 k 2024-03-18T01:09:46.230 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2024-03-18T01:09:46.727 INFO:teuthology.orchestra.run.smithi029.stdout:rbd-nbd-17.2.7-684.g6716c9f3.el9.x86_64.rpm 332 kB/s | 165 kB 00:00 2024-03-18T01:09:46.727 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2024-03-18T01:09:46.727 INFO:teuthology.orchestra.run.smithi029.stdout:Total 331 kB/s | 165 kB 00:00 2024-03-18T01:09:46.728 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2024-03-18T01:09:46.736 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2024-03-18T01:09:46.736 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2024-03-18T01:09:46.795 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2024-03-18T01:09:46.795 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2024-03-18T01:09:46.946 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2024-03-18T01:09:47.056 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : rbd-nbd-2:17.2.7-684.g6716c9f3.el9.x86_64 1/1 2024-03-18T01:09:47.532 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: rbd-nbd-2:17.2.7-684.g6716c9f3.el9.x86_64 1/1 2024-03-18T01:09:47.833 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : rbd-nbd-2:17.2.7-684.g6716c9f3.el9.x86_64 1/1 2024-03-18T01:09:47.833 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-18T01:09:47.833 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2024-03-18T01:09:47.833 INFO:teuthology.orchestra.run.smithi029.stdout: rbd-nbd-2:17.2.7-684.g6716c9f3.el9.x86_64 2024-03-18T01:09:47.833 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-18T01:09:47.834 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2024-03-18T01:09:48.120 DEBUG:teuthology.parallel:result is None 2024-03-18T01:09:48.121 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-18T01:09:48.121 INFO:teuthology.packaging:ref: None 2024-03-18T01:09:48.121 INFO:teuthology.packaging:tag: None 2024-03-18T01:09:48.121 INFO:teuthology.packaging:branch: quincy 2024-03-18T01:09:48.121 INFO:teuthology.packaging:sha1: 3d78f5ba07bf18880172c07d4d7b4ac00c9d30af 2024-03-18T01:09:48.121 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=quincy 2024-03-18T01:09:48.531 DEBUG:teuthology.orchestra.run.smithi029:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-18T01:09:48.553 INFO:teuthology.orchestra.run.smithi029.stdout:17.2.7-684.g6716c9f3.el9 2024-03-18T01:09:48.553 INFO:teuthology.packaging:The installed version of ceph is 17.2.7-684.g6716c9f3.el9 2024-03-18T01:09:48.553 INFO:teuthology.task.install:The correct ceph version 17.2.7-684.g6716c9f3 is installed. 2024-03-18T01:09:48.555 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-18T01:09:48.555 INFO:teuthology.packaging:ref: None 2024-03-18T01:09:48.555 INFO:teuthology.packaging:tag: None 2024-03-18T01:09:48.556 INFO:teuthology.packaging:branch: quincy 2024-03-18T01:09:48.556 INFO:teuthology.packaging:sha1: 3d78f5ba07bf18880172c07d4d7b4ac00c9d30af 2024-03-18T01:09:48.556 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=quincy 2024-03-18T01:09:48.876 DEBUG:teuthology.orchestra.run.smithi090:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-18T01:09:48.901 INFO:teuthology.orchestra.run.smithi090.stdout:17.2.7-684.g6716c9f3.el9 2024-03-18T01:09:48.902 INFO:teuthology.packaging:The installed version of ceph is 17.2.7-684.g6716c9f3.el9 2024-03-18T01:09:48.902 INFO:teuthology.task.install:The correct ceph version 17.2.7-684.g6716c9f3 is installed. 2024-03-18T01:09:48.904 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-18T01:09:48.904 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-03-18T01:09:48.904 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-18T01:09:48.937 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-03-18T01:09:48.937 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-18T01:09:48.973 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-18T01:09:48.974 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-03-18T01:09:48.974 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd of=/usr/bin/daemon-helper 2024-03-18T01:09:49.008 DEBUG:teuthology.orchestra.run.smithi029:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-18T01:09:49.079 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-03-18T01:09:49.079 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd of=/usr/bin/daemon-helper 2024-03-18T01:09:49.110 DEBUG:teuthology.orchestra.run.smithi090:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-18T01:09:49.181 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-18T01:09:49.181 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-03-18T01:09:49.181 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-18T01:09:49.212 DEBUG:teuthology.orchestra.run.smithi029:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-18T01:09:49.283 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-03-18T01:09:49.283 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-18T01:09:49.314 DEBUG:teuthology.orchestra.run.smithi090:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-18T01:09:49.382 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-18T01:09:49.382 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-03-18T01:09:49.382 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd of=/usr/bin/stdin-killer 2024-03-18T01:09:49.413 DEBUG:teuthology.orchestra.run.smithi029:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-18T01:09:49.485 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-03-18T01:09:49.644 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd of=/usr/bin/stdin-killer 2024-03-18T01:09:49.673 DEBUG:teuthology.orchestra.run.smithi090:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-18T01:09:49.742 INFO:teuthology.run_tasks:Running task print... 2024-03-18T01:09:49.759 INFO:teuthology.task.print:**** done install task... 2024-03-18T01:09:49.759 INFO:teuthology.run_tasks:Running task cephadm... 2024-03-18T01:09:49.858 INFO:tasks.cephadm:Config: {'cephadm_branch': 'quincy', 'cephadm_git_url': 'https://github.com/ceph/ceph', 'conf': {'osd': {'osd_class_default_list': '*', 'osd_class_load_list': '*', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}, 'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'mon pg warn min per osd': 0}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}}, 'image': 'quay.ceph.io/ceph-ci/ceph:quincy', 'roleless': True, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'mons down, quorum', 'OSD_DOWN', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'MON_DOWN'], 'sha1': '3d78f5ba07bf18880172c07d4d7b4ac00c9d30af'} 2024-03-18T01:09:49.859 INFO:tasks.cephadm:Cluster image is quay.ceph.io/ceph-ci/ceph:quincy 2024-03-18T01:09:49.859 INFO:tasks.cephadm:Cluster fsid is 37f51182-e4c4-11ee-95c9-87774f69a715 2024-03-18T01:09:49.859 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-03-18T01:09:49.859 INFO:tasks.cephadm:No mon roles; fabricating mons 2024-03-18T01:09:49.859 INFO:tasks.cephadm:Monitor IPs: {'mon.smithi029': '172.21.15.29', 'mon.smithi090': '172.21.15.90'} 2024-03-18T01:09:49.860 INFO:tasks.cephadm:Normalizing hostnames... 2024-03-18T01:09:49.860 DEBUG:teuthology.orchestra.run.smithi029:> sudo hostname $(hostname -s) 2024-03-18T01:09:49.892 DEBUG:teuthology.orchestra.run.smithi090:> sudo hostname $(hostname -s) 2024-03-18T01:09:49.919 INFO:tasks.cephadm:Downloading cephadm (repo https://github.com/ceph/ceph ref quincy)... 2024-03-18T01:09:49.919 DEBUG:teuthology.orchestra.run.smithi029:> curl --silent https://raw.githubusercontent.com/ceph/ceph/quincy/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-18T01:09:50.159 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r--. 1 ubuntu ubuntu 375153 Mar 18 01:09 /home/ubuntu/cephtest/cephadm 2024-03-18T01:09:50.160 DEBUG:teuthology.orchestra.run.smithi090:> curl --silent https://raw.githubusercontent.com/ceph/ceph/quincy/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-18T01:09:50.363 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r--. 1 ubuntu ubuntu 375153 Mar 18 01:09 /home/ubuntu/cephtest/cephadm 2024-03-18T01:09:50.364 DEBUG:teuthology.orchestra.run.smithi029:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-18T01:09:50.382 DEBUG:teuthology.orchestra.run.smithi090:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-18T01:09:50.408 INFO:tasks.cephadm:Pulling image quay.ceph.io/ceph-ci/ceph:quincy on all hosts... 2024-03-18T01:09:50.408 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy pull 2024-03-18T01:09:50.425 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy pull 2024-03-18T01:09:51.403 INFO:teuthology.orchestra.run.smithi090.stderr:Pulling container image quay.ceph.io/ceph-ci/ceph:quincy... 2024-03-18T01:09:51.408 INFO:teuthology.orchestra.run.smithi029.stderr:Pulling container image quay.ceph.io/ceph-ci/ceph:quincy... 2024-03-18T01:10:34.845 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-03-18T01:10:34.845 INFO:teuthology.orchestra.run.smithi090.stdout: "ceph_version": "ceph version 17.2.7-684-g6716c9f3 (6716c9f3976cf9fc2fd546e9f1182c2d4421b451) quincy (stable)", 2024-03-18T01:10:34.845 INFO:teuthology.orchestra.run.smithi090.stdout: "image_id": "19857db3290c58169e44a8aec8ef9646981a77ffd5b146ecef1b6faf59d6c3ab", 2024-03-18T01:10:34.845 INFO:teuthology.orchestra.run.smithi090.stdout: "repo_digests": [ 2024-03-18T01:10:34.845 INFO:teuthology.orchestra.run.smithi090.stdout: "quay.ceph.io/ceph-ci/ceph@sha256:a75c953aa7dcb8597f235a84417a8be589209dba39db0305cebbb10d212afaf9" 2024-03-18T01:10:34.845 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-03-18T01:10:34.845 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-03-18T01:10:40.731 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-03-18T01:10:40.731 INFO:teuthology.orchestra.run.smithi029.stdout: "ceph_version": "ceph version 17.2.7-684-g6716c9f3 (6716c9f3976cf9fc2fd546e9f1182c2d4421b451) quincy (stable)", 2024-03-18T01:10:40.732 INFO:teuthology.orchestra.run.smithi029.stdout: "image_id": "19857db3290c58169e44a8aec8ef9646981a77ffd5b146ecef1b6faf59d6c3ab", 2024-03-18T01:10:40.732 INFO:teuthology.orchestra.run.smithi029.stdout: "repo_digests": [ 2024-03-18T01:10:40.732 INFO:teuthology.orchestra.run.smithi029.stdout: "quay.ceph.io/ceph-ci/ceph@sha256:a75c953aa7dcb8597f235a84417a8be589209dba39db0305cebbb10d212afaf9" 2024-03-18T01:10:40.732 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-18T01:10:40.732 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-03-18T01:10:40.751 DEBUG:teuthology.orchestra.run.smithi029:> sudo mkdir -p /etc/ceph 2024-03-18T01:10:40.786 DEBUG:teuthology.orchestra.run.smithi090:> sudo mkdir -p /etc/ceph 2024-03-18T01:10:40.822 DEBUG:teuthology.orchestra.run.smithi029:> sudo chmod 777 /etc/ceph 2024-03-18T01:10:40.853 DEBUG:teuthology.orchestra.run.smithi090:> sudo chmod 777 /etc/ceph 2024-03-18T01:10:40.892 INFO:tasks.cephadm:Writing seed config... 2024-03-18T01:10:40.893 INFO:tasks.cephadm: override: [osd] osd_class_default_list = * 2024-03-18T01:10:40.893 INFO:tasks.cephadm: override: [osd] osd_class_load_list = * 2024-03-18T01:10:40.893 INFO:tasks.cephadm: override: [osd] bdev async discard = True 2024-03-18T01:10:40.893 INFO:tasks.cephadm: override: [osd] bdev enable discard = True 2024-03-18T01:10:40.893 INFO:tasks.cephadm: override: [osd] bluestore allocator = bitmap 2024-03-18T01:10:40.893 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2024-03-18T01:10:40.894 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2024-03-18T01:10:40.894 INFO:tasks.cephadm: override: [osd] debug bluefs = 1/20 2024-03-18T01:10:40.894 INFO:tasks.cephadm: override: [osd] debug bluestore = 1/20 2024-03-18T01:10:40.894 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-03-18T01:10:40.894 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-03-18T01:10:40.894 INFO:tasks.cephadm: override: [osd] debug rocksdb = 4/10 2024-03-18T01:10:40.894 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2024-03-18T01:10:40.894 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2024-03-18T01:10:40.894 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2024-03-18T01:10:40.894 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2024-03-18T01:10:40.894 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2024-03-18T01:10:40.894 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2024-03-18T01:10:40.894 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2024-03-18T01:10:40.895 INFO:tasks.cephadm: override: [client] debug client = 20 2024-03-18T01:10:40.895 INFO:tasks.cephadm: override: [client] debug ms = 1 2024-03-18T01:10:40.895 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2024-03-18T01:10:40.895 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2024-03-18T01:10:40.895 INFO:tasks.cephadm: override: [global] mon pg warn min per osd = 0 2024-03-18T01:10:40.895 INFO:tasks.cephadm: override: [mds] debug mds = 20 2024-03-18T01:10:40.895 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2024-03-18T01:10:40.895 INFO:tasks.cephadm: override: [mds] debug ms = 1 2024-03-18T01:10:40.895 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2024-03-18T01:10:40.895 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2024-03-18T01:10:40.895 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2024-03-18T01:10:40.895 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2024-03-18T01:10:40.895 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2024-03-18T01:10:40.895 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2024-03-18T01:10:40.895 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2024-03-18T01:10:40.895 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-03-18T01:10:40.896 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-03-18T01:10:40.896 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-03-18T01:10:40.896 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-03-18T01:10:40.896 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-03-18T01:10:40.896 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2024-03-18T01:10:40.896 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-03-18T01:10:40.897 DEBUG:teuthology.orchestra.run.smithi029:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-03-18T01:10:40.911 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 37f51182-e4c4-11ee-95c9-87774f69a715 mon pg warn min per osd = 0 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = true bdev debug aio = true osd sloppy crc = true osd_class_default_list = * osd_class_load_list = * bdev async discard = True bdev enable discard = True bluestore allocator = bitmap bluestore block size = 96636764160 bluestore fsck on mount = True debug bluefs = 1/20 debug bluestore = 1/20 debug ms = 1 debug osd = 20 debug rocksdb = 4/10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2024-03-18T01:10:40.911 DEBUG:teuthology.orchestra.run.smithi029:mon.smithi029> sudo journalctl -f -n 0 -u ceph-37f51182-e4c4-11ee-95c9-87774f69a715@mon.smithi029.service 2024-03-18T01:10:40.953 INFO:tasks.cephadm:Bootstrapping... 2024-03-18T01:10:40.953 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy -v bootstrap --fsid 37f51182-e4c4-11ee-95c9-87774f69a715 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-ip 172.21.15.29 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-03-18T01:10:41.164 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2024-03-18T01:10:41.164 INFO:teuthology.orchestra.run.smithi029.stdout:cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph:quincy', '-v', 'bootstrap', '--fsid', '37f51182-e4c4-11ee-95c9-87774f69a715', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-ip', '172.21.15.29', '--skip-admin-label'] 2024-03-18T01:10:41.191 INFO:teuthology.orchestra.run.smithi029.stdout:/bin/podman: stdout 4.9.3-dev 2024-03-18T01:10:41.192 INFO:teuthology.orchestra.run.smithi029.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-03-18T01:10:41.192 INFO:teuthology.orchestra.run.smithi029.stdout:Verifying podman|docker is present... 2024-03-18T01:10:41.216 INFO:teuthology.orchestra.run.smithi029.stdout:/bin/podman: stdout 4.9.3-dev 2024-03-18T01:10:41.216 INFO:teuthology.orchestra.run.smithi029.stdout:Verifying lvm2 is present... 2024-03-18T01:10:41.216 INFO:teuthology.orchestra.run.smithi029.stdout:Verifying time synchronization is in place... 2024-03-18T01:10:41.224 INFO:teuthology.orchestra.run.smithi029.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-03-18T01:10:41.224 INFO:teuthology.orchestra.run.smithi029.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-03-18T01:10:41.230 INFO:teuthology.orchestra.run.smithi029.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-03-18T01:10:41.230 INFO:teuthology.orchestra.run.smithi029.stdout:systemctl: stdout inactive 2024-03-18T01:10:41.236 INFO:teuthology.orchestra.run.smithi029.stdout:systemctl: stdout enabled 2024-03-18T01:10:41.242 INFO:teuthology.orchestra.run.smithi029.stdout:systemctl: stdout active 2024-03-18T01:10:41.242 INFO:teuthology.orchestra.run.smithi029.stdout:Unit chronyd.service is enabled and running 2024-03-18T01:10:41.242 INFO:teuthology.orchestra.run.smithi029.stdout:Repeating the final host check... 2024-03-18T01:10:41.264 INFO:teuthology.orchestra.run.smithi029.stdout:/bin/podman: stdout 4.9.3-dev 2024-03-18T01:10:41.265 INFO:teuthology.orchestra.run.smithi029.stdout:podman (/bin/podman) version 4.9.3 is present 2024-03-18T01:10:41.265 INFO:teuthology.orchestra.run.smithi029.stdout:systemctl is present 2024-03-18T01:10:41.265 INFO:teuthology.orchestra.run.smithi029.stdout:lvcreate is present 2024-03-18T01:10:41.271 INFO:teuthology.orchestra.run.smithi029.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-03-18T01:10:41.271 INFO:teuthology.orchestra.run.smithi029.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-03-18T01:10:41.277 INFO:teuthology.orchestra.run.smithi029.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-03-18T01:10:41.277 INFO:teuthology.orchestra.run.smithi029.stdout:systemctl: stdout inactive 2024-03-18T01:10:41.284 INFO:teuthology.orchestra.run.smithi029.stdout:systemctl: stdout enabled 2024-03-18T01:10:41.290 INFO:teuthology.orchestra.run.smithi029.stdout:systemctl: stdout active 2024-03-18T01:10:41.290 INFO:teuthology.orchestra.run.smithi029.stdout:Unit chronyd.service is enabled and running 2024-03-18T01:10:41.291 INFO:teuthology.orchestra.run.smithi029.stdout:Host looks OK 2024-03-18T01:10:41.291 INFO:teuthology.orchestra.run.smithi029.stdout:Cluster fsid: 37f51182-e4c4-11ee-95c9-87774f69a715 2024-03-18T01:10:41.291 INFO:teuthology.orchestra.run.smithi029.stdout:Acquiring lock 139687032103072 on /run/cephadm/37f51182-e4c4-11ee-95c9-87774f69a715.lock 2024-03-18T01:10:41.291 INFO:teuthology.orchestra.run.smithi029.stdout:Lock 139687032103072 acquired on /run/cephadm/37f51182-e4c4-11ee-95c9-87774f69a715.lock 2024-03-18T01:10:41.291 INFO:teuthology.orchestra.run.smithi029.stdout:Verifying IP 172.21.15.29 port 3300 ... 2024-03-18T01:10:41.291 INFO:teuthology.orchestra.run.smithi029.stdout:Verifying IP 172.21.15.29 port 6789 ... 2024-03-18T01:10:41.291 INFO:teuthology.orchestra.run.smithi029.stdout:Base mon IP(s) is [172.21.15.29:3300, 172.21.15.29:6789], mon addrv is [v2:172.21.15.29:3300,v1:172.21.15.29:6789] 2024-03-18T01:10:41.295 INFO:teuthology.orchestra.run.smithi029.stdout:/sbin/ip: stdout default via 172.21.15.254 dev ens1f0 proto dhcp src 172.21.15.29 metric 100 2024-03-18T01:10:41.295 INFO:teuthology.orchestra.run.smithi029.stdout:/sbin/ip: stdout 172.21.0.0/20 dev ens1f0 proto kernel scope link src 172.21.15.29 metric 100 2024-03-18T01:10:41.298 INFO:teuthology.orchestra.run.smithi029.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-03-18T01:10:41.298 INFO:teuthology.orchestra.run.smithi029.stdout:/sbin/ip: stdout fe80::/64 dev ens1f0 proto kernel metric 1024 pref medium 2024-03-18T01:10:41.298 INFO:teuthology.orchestra.run.smithi029.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev ens1f0 proto ra metric 100 pref medium 2024-03-18T01:10:41.300 INFO:teuthology.orchestra.run.smithi029.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-03-18T01:10:41.300 INFO:teuthology.orchestra.run.smithi029.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-03-18T01:10:41.300 INFO:teuthology.orchestra.run.smithi029.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-03-18T01:10:41.301 INFO:teuthology.orchestra.run.smithi029.stdout:/sbin/ip: stdout 4: ens1f0: mtu 1500 state UP qlen 1000 2024-03-18T01:10:41.301 INFO:teuthology.orchestra.run.smithi029.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:febd:125a/64 scope link noprefixroute 2024-03-18T01:10:41.301 INFO:teuthology.orchestra.run.smithi029.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-03-18T01:10:41.301 INFO:teuthology.orchestra.run.smithi029.stdout:Mon IP `172.21.15.29` is in CIDR network `172.21.0.0/20` 2024-03-18T01:10:41.301 INFO:teuthology.orchestra.run.smithi029.stdout:Mon IP `172.21.15.29` is in CIDR network `172.21.0.0/20` 2024-03-18T01:10:41.302 INFO:teuthology.orchestra.run.smithi029.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-03-18T01:10:41.302 INFO:teuthology.orchestra.run.smithi029.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-03-18T01:10:41.302 INFO:teuthology.orchestra.run.smithi029.stdout:Pulling container image quay.ceph.io/ceph-ci/ceph:quincy... 2024-03-18T01:10:41.819 INFO:teuthology.orchestra.run.smithi029.stdout:/bin/podman: stdout 19857db3290c58169e44a8aec8ef9646981a77ffd5b146ecef1b6faf59d6c3ab 2024-03-18T01:10:41.819 INFO:teuthology.orchestra.run.smithi029.stdout:/bin/podman: stderr Trying to pull quay.ceph.io/ceph-ci/ceph:quincy... 2024-03-18T01:10:41.820 INFO:teuthology.orchestra.run.smithi029.stdout:/bin/podman: stderr Getting image source signatures 2024-03-18T01:10:41.820 INFO:teuthology.orchestra.run.smithi029.stdout:/bin/podman: stderr Copying blob sha256:f58b4670ef4403d11d1c912aa26a4dbcf73447df5302e9dc66f6a7fc0d9d1e47 2024-03-18T01:10:41.820 INFO:teuthology.orchestra.run.smithi029.stdout:/bin/podman: stderr Copying blob sha256:dde26ec1da815cb53894018b523d9a5c6c7454b327da4e857899d556db304927 2024-03-18T01:10:41.820 INFO:teuthology.orchestra.run.smithi029.stdout:/bin/podman: stderr Copying config sha256:19857db3290c58169e44a8aec8ef9646981a77ffd5b146ecef1b6faf59d6c3ab 2024-03-18T01:10:41.820 INFO:teuthology.orchestra.run.smithi029.stdout:/bin/podman: stderr Writing manifest to image destination 2024-03-18T01:10:42.790 INFO:teuthology.orchestra.run.smithi029.stdout:ceph: stdout ceph version 17.2.7-684-g6716c9f3 (6716c9f3976cf9fc2fd546e9f1182c2d4421b451) quincy (stable) 2024-03-18T01:10:42.790 INFO:teuthology.orchestra.run.smithi029.stdout:Ceph version: ceph version 17.2.7-684-g6716c9f3 (6716c9f3976cf9fc2fd546e9f1182c2d4421b451) quincy (stable) 2024-03-18T01:10:42.790 INFO:teuthology.orchestra.run.smithi029.stdout:Extracting ceph user uid/gid from container image... 2024-03-18T01:10:43.828 INFO:teuthology.orchestra.run.smithi029.stdout:stat: stdout 167 167 2024-03-18T01:10:43.829 INFO:teuthology.orchestra.run.smithi029.stdout:Creating initial keys... 2024-03-18T01:10:44.742 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph-authtool: stdout AQCUlPdlR6xRCRAAuDMvZy/u2rTvuu2ceLNDbg== 2024-03-18T01:10:45.680 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph-authtool: stdout AQCVlPdlFcSJBRAAsI4vkB15CkGgZQaYxTwDVw== 2024-03-18T01:10:46.600 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph-authtool: stdout AQCWlPdlu0/kABAAVZcRjuCyO8w4COGnu4ONZA== 2024-03-18T01:10:46.601 INFO:teuthology.orchestra.run.smithi029.stdout:Creating initial monmap... 2024-03-18T01:10:47.541 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-18T01:10:47.541 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = octopus 2024-03-18T01:10:47.541 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to 37f51182-e4c4-11ee-95c9-87774f69a715 2024-03-18T01:10:47.541 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-18T01:10:47.542 INFO:teuthology.orchestra.run.smithi029.stdout:monmaptool for smithi029 [v2:172.21.15.29:3300,v1:172.21.15.29:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-18T01:10:47.542 INFO:teuthology.orchestra.run.smithi029.stdout:setting min_mon_release = octopus 2024-03-18T01:10:47.542 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/monmaptool: set fsid to 37f51182-e4c4-11ee-95c9-87774f69a715 2024-03-18T01:10:47.542 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-18T01:10:47.542 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-18T01:10:47.542 INFO:teuthology.orchestra.run.smithi029.stdout:Creating mon... 2024-03-18T01:10:48.595 INFO:teuthology.orchestra.run.smithi029.stdout:create mon.smithi029 on 2024-03-18T01:10:48.811 INFO:teuthology.orchestra.run.smithi029.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-03-18T01:10:48.954 INFO:teuthology.orchestra.run.smithi029.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-03-18T01:10:49.105 INFO:teuthology.orchestra.run.smithi029.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-37f51182-e4c4-11ee-95c9-87774f69a715.target → /etc/systemd/system/ceph-37f51182-e4c4-11ee-95c9-87774f69a715.target. 2024-03-18T01:10:49.105 INFO:teuthology.orchestra.run.smithi029.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-37f51182-e4c4-11ee-95c9-87774f69a715.target → /etc/systemd/system/ceph-37f51182-e4c4-11ee-95c9-87774f69a715.target. 2024-03-18T01:10:49.274 INFO:teuthology.orchestra.run.smithi029.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-37f51182-e4c4-11ee-95c9-87774f69a715@mon.smithi029 2024-03-18T01:10:49.274 INFO:teuthology.orchestra.run.smithi029.stdout:systemctl: stderr Failed to reset failed state of unit ceph-37f51182-e4c4-11ee-95c9-87774f69a715@mon.smithi029.service: Unit ceph-37f51182-e4c4-11ee-95c9-87774f69a715@mon.smithi029.service not loaded. 2024-03-18T01:10:49.433 INFO:teuthology.orchestra.run.smithi029.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-37f51182-e4c4-11ee-95c9-87774f69a715.target.wants/ceph-37f51182-e4c4-11ee-95c9-87774f69a715@mon.smithi029.service → /etc/systemd/system/ceph-37f51182-e4c4-11ee-95c9-87774f69a715@.service. 2024-03-18T01:10:49.949 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:10:49 smithi029 podman[28945]: 2024-03-18 01:10:49.887588704 +0000 UTC m=+0.335273722 container start 55d7394db9ec6a91d24d4dd09aa3f7a4106d02709dcb63312cf15f737cc1f547 (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, RELEASE=quincy-6716c9f, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True) 2024-03-18T01:10:49.949 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:10:49 smithi029 ceph-mon[28964]: mkfs 37f51182-e4c4-11ee-95c9-87774f69a715 2024-03-18T01:10:49.998 INFO:teuthology.orchestra.run.smithi029.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-18T01:10:49.999 INFO:teuthology.orchestra.run.smithi029.stdout:systemctl: stdout disabled 2024-03-18T01:10:50.005 INFO:teuthology.orchestra.run.smithi029.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-18T01:10:50.005 INFO:teuthology.orchestra.run.smithi029.stdout:systemctl: stdout inactive 2024-03-18T01:10:50.005 INFO:teuthology.orchestra.run.smithi029.stdout:firewalld.service is not enabled 2024-03-18T01:10:50.005 INFO:teuthology.orchestra.run.smithi029.stdout:Not possible to enable service . firewalld.service is not available 2024-03-18T01:10:50.006 INFO:teuthology.orchestra.run.smithi029.stdout:Waiting for mon to start... 2024-03-18T01:10:50.006 INFO:teuthology.orchestra.run.smithi029.stdout:Waiting for mon... 2024-03-18T01:10:50.214 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:10:49 smithi029 ceph-mon[28964]: mon.smithi029 is new leader, mons smithi029 in quorum (ranks 0) 2024-03-18T01:10:50.214 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:10:49 smithi029 bash[28945]: 55d7394db9ec6a91d24d4dd09aa3f7a4106d02709dcb63312cf15f737cc1f547 2024-03-18T01:10:50.214 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:10:49 smithi029 systemd[1]: Started Ceph mon.smithi029 for 37f51182-e4c4-11ee-95c9-87774f69a715. 2024-03-18T01:10:51.052 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout cluster: 2024-03-18T01:10:51.052 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout id: 37f51182-e4c4-11ee-95c9-87774f69a715 2024-03-18T01:10:51.052 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-03-18T01:10:51.052 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout 2024-03-18T01:10:51.052 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout services: 2024-03-18T01:10:51.052 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum smithi029 (age 0.555014s) 2024-03-18T01:10:51.052 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-03-18T01:10:51.052 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-03-18T01:10:51.052 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout 2024-03-18T01:10:51.052 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout data: 2024-03-18T01:10:51.052 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-03-18T01:10:51.052 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-03-18T01:10:51.053 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-03-18T01:10:51.053 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout pgs: 2024-03-18T01:10:51.053 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout 2024-03-18T01:10:51.053 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:10:50.421+0000 7fa384d4b700 1 Processor -- start 2024-03-18T01:10:51.053 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:10:50.422+0000 7fa384d4b700 1 -- start start 2024-03-18T01:10:51.053 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:10:50.422+0000 7fa384d4b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa380102730 0x7fa380102b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:10:51.053 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:10:50.422+0000 7fa384d4b700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa380103080 con 0x7fa380102730 2024-03-18T01:10:51.053 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:10:50.422+0000 7fa37e59c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa380102730 0x7fa380102b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:10:51.053 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:10:50.422+0000 7fa37e59c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa380102730 0x7fa380102b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39170/0 (socket says 172.21.15.29:39170) 2024-03-18T01:10:51.053 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:10:50.422+0000 7fa37e59c700 1 -- 172.21.15.29:0/2198275737 learned_addr learned my addr 172.21.15.29:0/2198275737 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-18T01:10:51.053 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:10:50.423+0000 7fa37e59c700 1 -- 172.21.15.29:0/2198275737 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3801031c0 con 0x7fa380102730 2024-03-18T01:10:51.053 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:10:50.423+0000 7fa37e59c700 1 --2- 172.21.15.29:0/2198275737 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa380102730 0x7fa380102b40 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fa3740098e0 tx=0x7fa374009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef7438c7b0aa47ec server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:10:51.053 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:10:50.423+0000 7fa37d59a700 1 -- 172.21.15.29:0/2198275737 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa37400b860 con 0x7fa380102730 2024-03-18T01:10:51.053 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:10:50.423+0000 7fa37d59a700 1 -- 172.21.15.29:0/2198275737 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa37400b9c0 con 0x7fa380102730 2024-03-18T01:10:51.054 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:10:50.423+0000 7fa37d59a700 1 -- 172.21.15.29:0/2198275737 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa37400bc90 con 0x7fa380102730 2024-03-18T01:10:51.054 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:10:50.424+0000 7fa384d4b700 1 -- 172.21.15.29:0/2198275737 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa380102730 msgr2=0x7fa380102b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:10:51.054 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:10:50.424+0000 7fa384d4b700 1 --2- 172.21.15.29:0/2198275737 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa380102730 0x7fa380102b40 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fa3740098e0 tx=0x7fa374009250 comp rx=0 tx=0).stop 2024-03-18T01:10:51.054 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:10:50.424+0000 7fa384d4b700 1 -- 172.21.15.29:0/2198275737 shutdown_connections 2024-03-18T01:10:51.054 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:10:50.424+0000 7fa384d4b700 1 --2- 172.21.15.29:0/2198275737 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa380102730 0x7fa380102b40 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:10:51.054 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:10:50.424+0000 7fa384d4b700 1 -- 172.21.15.29:0/2198275737 >> 172.21.15.29:0/2198275737 conn(0x7fa3800fdf10 msgr2=0x7fa380100320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:10:51.054 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:10:50.424+0000 7fa384d4b700 1 -- 172.21.15.29:0/2198275737 shutdown_connections 2024-03-18T01:10:51.054 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:10:50.424+0000 7fa384d4b700 1 -- 172.21.15.29:0/2198275737 wait complete. 2024-03-18T01:10:51.054 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:10:50.424+0000 7fa384d4b700 1 Processor -- start 2024-03-18T01:10:51.054 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:10:50.424+0000 7fa384d4b700 1 -- start start 2024-03-18T01:10:51.054 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:10:50.424+0000 7fa384d4b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa380102730 0x7fa380078bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:10:51.054 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:10:50.424+0000 7fa384d4b700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa380079110 con 0x7fa380102730 2024-03-18T01:10:51.054 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:10:50.424+0000 7fa37e59c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa380102730 0x7fa380078bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:10:51.055 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:10:50.424+0000 7fa37e59c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa380102730 0x7fa380078bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39180/0 (socket says 172.21.15.29:39180) 2024-03-18T01:10:51.055 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:10:50.424+0000 7fa37e59c700 1 -- 172.21.15.29:0/1550820376 learned_addr learned my addr 172.21.15.29:0/1550820376 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-18T01:10:51.055 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:10:50.425+0000 7fa37e59c700 1 -- 172.21.15.29:0/1550820376 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3740094f0 con 0x7fa380102730 2024-03-18T01:10:51.055 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:10:50.425+0000 7fa37e59c700 1 --2- 172.21.15.29:0/1550820376 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa380102730 0x7fa380078bd0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fa374009290 tx=0x7fa374016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:10:51.055 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:10:50.425+0000 7fa36effd700 1 -- 172.21.15.29:0/1550820376 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa37401c5c0 con 0x7fa380102730 2024-03-18T01:10:51.055 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:10:50.425+0000 7fa36effd700 1 -- 172.21.15.29:0/1550820376 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa37401c740 con 0x7fa380102730 2024-03-18T01:10:51.055 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:10:50.425+0000 7fa384d4b700 1 -- 172.21.15.29:0/1550820376 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa380077180 con 0x7fa380102730 2024-03-18T01:10:51.055 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:10:50.425+0000 7fa36effd700 1 -- 172.21.15.29:0/1550820376 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa37401ca50 con 0x7fa380102730 2024-03-18T01:10:51.055 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:10:50.425+0000 7fa384d4b700 1 -- 172.21.15.29:0/1550820376 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa380077670 con 0x7fa380102730 2024-03-18T01:10:51.055 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:10:50.425+0000 7fa36effd700 1 -- 172.21.15.29:0/1550820376 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7fa37401a450 con 0x7fa380102730 2024-03-18T01:10:51.055 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:10:50.425+0000 7fa36effd700 1 -- 172.21.15.29:0/1550820376 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7fa37402e790 con 0x7fa380102730 2024-03-18T01:10:51.055 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:10:50.426+0000 7fa384d4b700 1 -- 172.21.15.29:0/1550820376 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa38005f5f0 con 0x7fa380102730 2024-03-18T01:10:51.055 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:10:50.428+0000 7fa36effd700 1 -- 172.21.15.29:0/1550820376 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7fa37401ce30 con 0x7fa380102730 2024-03-18T01:10:51.055 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:10:50.500+0000 7fa384d4b700 1 -- 172.21.15.29:0/1550820376 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "status"} v 0) v1 -- 0x7fa38010a2f0 con 0x7fa380102730 2024-03-18T01:10:51.055 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:10:50.500+0000 7fa36effd700 1 -- 172.21.15.29:0/1550820376 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) v1 ==== 54+0+325 (secure 0 0 0) 0x7fa37401a840 con 0x7fa380102730 2024-03-18T01:10:51.056 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:10:50.501+0000 7fa384d4b700 1 -- 172.21.15.29:0/1550820376 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa380102730 msgr2=0x7fa380078bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:10:51.056 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:10:50.501+0000 7fa384d4b700 1 --2- 172.21.15.29:0/1550820376 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa380102730 0x7fa380078bd0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fa374009290 tx=0x7fa374016000 comp rx=0 tx=0).stop 2024-03-18T01:10:51.056 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:10:50.501+0000 7fa384d4b700 1 -- 172.21.15.29:0/1550820376 shutdown_connections 2024-03-18T01:10:51.056 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:10:50.501+0000 7fa384d4b700 1 --2- 172.21.15.29:0/1550820376 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa380102730 0x7fa380078bd0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:10:51.056 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:10:50.501+0000 7fa384d4b700 1 -- 172.21.15.29:0/1550820376 >> 172.21.15.29:0/1550820376 conn(0x7fa3800fdf10 msgr2=0x7fa380105200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:10:51.056 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:10:50.501+0000 7fa384d4b700 1 -- 172.21.15.29:0/1550820376 shutdown_connections 2024-03-18T01:10:51.056 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:10:50.501+0000 7fa384d4b700 1 -- 172.21.15.29:0/1550820376 wait complete. 2024-03-18T01:10:51.056 INFO:teuthology.orchestra.run.smithi029.stdout:mon is available 2024-03-18T01:10:51.056 INFO:teuthology.orchestra.run.smithi029.stdout:Assimilating anything we can from ceph.conf... 2024-03-18T01:10:51.068 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:10:50 smithi029 ceph-mon[28964]: mon.smithi029 is new leader, mons smithi029 in quorum (ranks 0) 2024-03-18T01:10:51.069 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:10:50 smithi029 ceph-mon[28964]: monmap e1: 1 mons at {smithi029=[v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-18T01:10:51.069 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:10:50 smithi029 ceph-mon[28964]: fsmap 2024-03-18T01:10:51.069 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:10:50 smithi029 ceph-mon[28964]: osdmap e1: 0 total, 0 up, 0 in 2024-03-18T01:10:51.069 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:10:50 smithi029 ceph-mon[28964]: mgrmap e1: no daemons active 2024-03-18T01:10:51.069 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:10:50 smithi029 ceph-mon[28964]: from='client.? 172.21.15.29:0/1550820376' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-03-18T01:10:52.123 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:10:51 smithi029 ceph-mon[28964]: from='client.? 172.21.15.29:0/1088878034' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-03-18T01:10:52.124 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:10:51 smithi029 ceph-mon[28964]: from='client.? 172.21.15.29:0/1088878034' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-03-18T01:10:52.134 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout 2024-03-18T01:10:52.134 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout [global] 2024-03-18T01:10:52.134 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout fsid = 37f51182-e4c4-11ee-95c9-87774f69a715 2024-03-18T01:10:52.134 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.29:3300,v1:172.21.15.29:6789] 2024-03-18T01:10:52.135 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-03-18T01:10:52.135 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-03-18T01:10:52.135 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-03-18T01:10:52.135 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-03-18T01:10:52.135 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout 2024-03-18T01:10:52.135 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout [mgr] 2024-03-18T01:10:52.135 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-03-18T01:10:52.135 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout 2024-03-18T01:10:52.135 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout [osd] 2024-03-18T01:10:52.135 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-03-18T01:10:52.135 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-03-18T01:10:52.135 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-03-18T01:10:52.135 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-03-18T01:10:52.135 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-03-18T01:10:52.135 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:10:51.469+0000 7fa292bc4700 1 Processor -- start 2024-03-18T01:10:52.136 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:10:51.469+0000 7fa292bc4700 1 -- start start 2024-03-18T01:10:52.136 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:10:51.470+0000 7fa292bc4700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa28c102730 0x7fa28c102b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:10:52.136 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:10:51.470+0000 7fa292bc4700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa28c103080 con 0x7fa28c102730 2024-03-18T01:10:52.136 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:10:51.470+0000 7fa290960700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa28c102730 0x7fa28c102b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:10:52.136 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:10:51.470+0000 7fa290960700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa28c102730 0x7fa28c102b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39192/0 (socket says 172.21.15.29:39192) 2024-03-18T01:10:52.136 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:10:51.470+0000 7fa290960700 1 -- 172.21.15.29:0/1748254058 learned_addr learned my addr 172.21.15.29:0/1748254058 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-18T01:10:52.136 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:10:51.470+0000 7fa290960700 1 -- 172.21.15.29:0/1748254058 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa28c1031c0 con 0x7fa28c102730 2024-03-18T01:10:52.136 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:10:51.470+0000 7fa290960700 1 --2- 172.21.15.29:0/1748254058 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa28c102730 0x7fa28c102b40 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7fa2800098e0 tx=0x7fa280009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba050bca5a80d409 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:10:52.136 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:10:51.470+0000 7fa28b7fe700 1 -- 172.21.15.29:0/1748254058 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa28000b860 con 0x7fa28c102730 2024-03-18T01:10:52.136 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:10:51.471+0000 7fa28b7fe700 1 -- 172.21.15.29:0/1748254058 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa28000b9c0 con 0x7fa28c102730 2024-03-18T01:10:52.136 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:10:51.471+0000 7fa28b7fe700 1 -- 172.21.15.29:0/1748254058 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa28000bc90 con 0x7fa28c102730 2024-03-18T01:10:52.136 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:10:51.471+0000 7fa292bc4700 1 -- 172.21.15.29:0/1748254058 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa28c102730 msgr2=0x7fa28c102b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:10:52.136 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:10:51.471+0000 7fa292bc4700 1 --2- 172.21.15.29:0/1748254058 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa28c102730 0x7fa28c102b40 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7fa2800098e0 tx=0x7fa280009250 comp rx=0 tx=0).stop 2024-03-18T01:10:52.136 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:10:51.471+0000 7fa292bc4700 1 -- 172.21.15.29:0/1748254058 shutdown_connections 2024-03-18T01:10:52.137 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:10:51.471+0000 7fa292bc4700 1 --2- 172.21.15.29:0/1748254058 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa28c102730 0x7fa28c102b40 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:10:52.137 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:10:51.471+0000 7fa292bc4700 1 -- 172.21.15.29:0/1748254058 >> 172.21.15.29:0/1748254058 conn(0x7fa28c0fdf10 msgr2=0x7fa28c100320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:10:52.137 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:10:51.471+0000 7fa292bc4700 1 -- 172.21.15.29:0/1748254058 shutdown_connections 2024-03-18T01:10:52.137 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:10:51.471+0000 7fa292bc4700 1 -- 172.21.15.29:0/1748254058 wait complete. 2024-03-18T01:10:52.137 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:10:51.472+0000 7fa292bc4700 1 Processor -- start 2024-03-18T01:10:52.137 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:10:51.472+0000 7fa292bc4700 1 -- start start 2024-03-18T01:10:52.137 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:10:51.472+0000 7fa292bc4700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa28c102730 0x7fa28c110e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:10:52.137 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:10:51.472+0000 7fa292bc4700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa28c111380 con 0x7fa28c102730 2024-03-18T01:10:52.137 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:10:51.472+0000 7fa290960700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa28c102730 0x7fa28c110e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:10:52.137 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:10:51.472+0000 7fa290960700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa28c102730 0x7fa28c110e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39208/0 (socket says 172.21.15.29:39208) 2024-03-18T01:10:52.137 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:10:51.472+0000 7fa290960700 1 -- 172.21.15.29:0/1088878034 learned_addr learned my addr 172.21.15.29:0/1088878034 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-18T01:10:52.137 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:10:51.472+0000 7fa290960700 1 -- 172.21.15.29:0/1088878034 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2800094f0 con 0x7fa28c102730 2024-03-18T01:10:52.137 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:10:51.472+0000 7fa290960700 1 --2- 172.21.15.29:0/1088878034 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa28c102730 0x7fa28c110e40 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7fa280009290 tx=0x7fa280016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:10:52.137 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:10:51.473+0000 7fa2897fa700 1 -- 172.21.15.29:0/1088878034 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa28001c5c0 con 0x7fa28c102730 2024-03-18T01:10:52.138 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:10:51.473+0000 7fa2897fa700 1 -- 172.21.15.29:0/1088878034 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa28001c740 con 0x7fa28c102730 2024-03-18T01:10:52.138 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:10:51.473+0000 7fa292bc4700 1 -- 172.21.15.29:0/1088878034 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa28c111580 con 0x7fa28c102730 2024-03-18T01:10:52.138 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:10:51.473+0000 7fa2897fa700 1 -- 172.21.15.29:0/1088878034 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa28001ca50 con 0x7fa28c102730 2024-03-18T01:10:52.138 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:10:51.473+0000 7fa292bc4700 1 -- 172.21.15.29:0/1088878034 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa28c111fb0 con 0x7fa28c102730 2024-03-18T01:10:52.138 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:10:51.473+0000 7fa2897fa700 1 -- 172.21.15.29:0/1088878034 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7fa28001a450 con 0x7fa28c102730 2024-03-18T01:10:52.138 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:10:51.473+0000 7fa2897fa700 1 -- 172.21.15.29:0/1088878034 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7fa28002e790 con 0x7fa28c102730 2024-03-18T01:10:52.138 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:10:51.474+0000 7fa292bc4700 1 -- 172.21.15.29:0/1088878034 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa28c05f5f0 con 0x7fa28c102730 2024-03-18T01:10:52.138 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:10:51.476+0000 7fa2897fa700 1 -- 172.21.15.29:0/1088878034 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7fa28001ce30 con 0x7fa28c102730 2024-03-18T01:10:52.138 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:10:51.545+0000 7fa292bc4700 1 -- 172.21.15.29:0/1088878034 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7fa28c103c60 con 0x7fa28c102730 2024-03-18T01:10:52.138 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:10:51.549+0000 7fa2897fa700 1 -- 172.21.15.29:0/1088878034 <== mon.0 v2:172.21.15.29:3300/0 7 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7fa28002e9a0 con 0x7fa28c102730 2024-03-18T01:10:52.138 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:10:51.549+0000 7fa2897fa700 1 -- 172.21.15.29:0/1088878034 <== mon.0 v2:172.21.15.29:3300/0 8 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) v1 ==== 70+0+429 (secure 0 0 0) 0x7fa280037780 con 0x7fa28c102730 2024-03-18T01:10:52.138 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:10:51.549+0000 7fa292bc4700 1 -- 172.21.15.29:0/1088878034 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa28c102730 msgr2=0x7fa28c110e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:10:52.138 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:10:51.549+0000 7fa292bc4700 1 --2- 172.21.15.29:0/1088878034 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa28c102730 0x7fa28c110e40 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7fa280009290 tx=0x7fa280016000 comp rx=0 tx=0).stop 2024-03-18T01:10:52.138 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:10:51.550+0000 7fa292bc4700 1 -- 172.21.15.29:0/1088878034 shutdown_connections 2024-03-18T01:10:52.139 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:10:51.550+0000 7fa292bc4700 1 --2- 172.21.15.29:0/1088878034 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa28c102730 0x7fa28c110e40 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:10:52.139 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:10:51.550+0000 7fa292bc4700 1 -- 172.21.15.29:0/1088878034 >> 172.21.15.29:0/1088878034 conn(0x7fa28c0fdf10 msgr2=0x7fa28c105200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:10:52.139 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:10:51.550+0000 7fa292bc4700 1 -- 172.21.15.29:0/1088878034 shutdown_connections 2024-03-18T01:10:52.139 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:10:51.550+0000 7fa292bc4700 1 -- 172.21.15.29:0/1088878034 wait complete. 2024-03-18T01:10:52.139 INFO:teuthology.orchestra.run.smithi029.stdout:Generating new minimal ceph.conf... 2024-03-18T01:10:53.188 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:10:52 smithi029 ceph-mon[28964]: from='client.? 172.21.15.29:0/958991352' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:10:53.196 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:10:52.553+0000 7fcd64449700 1 Processor -- start 2024-03-18T01:10:53.197 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:10:52.553+0000 7fcd64449700 1 -- start start 2024-03-18T01:10:53.197 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:10:52.554+0000 7fcd64449700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcd5c076bc0 0x7fcd5c105310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:10:53.197 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:10:52.554+0000 7fcd64449700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd5c105850 con 0x7fcd5c076bc0 2024-03-18T01:10:53.197 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:10:52.554+0000 7fcd621e5700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcd5c076bc0 0x7fcd5c105310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:10:53.197 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:10:52.554+0000 7fcd621e5700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcd5c076bc0 0x7fcd5c105310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51830/0 (socket says 172.21.15.29:51830) 2024-03-18T01:10:53.197 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:10:52.554+0000 7fcd621e5700 1 -- 172.21.15.29:0/2546815462 learned_addr learned my addr 172.21.15.29:0/2546815462 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-18T01:10:53.197 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:10:52.554+0000 7fcd621e5700 1 -- 172.21.15.29:0/2546815462 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd5c105990 con 0x7fcd5c076bc0 2024-03-18T01:10:53.197 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:10:52.554+0000 7fcd621e5700 1 --2- 172.21.15.29:0/2546815462 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcd5c076bc0 0x7fcd5c105310 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fcd580098e0 tx=0x7fcd58009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=68b72638fd81eb22 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:10:53.197 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:10:52.554+0000 7fcd611e3700 1 -- 172.21.15.29:0/2546815462 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcd5800b860 con 0x7fcd5c076bc0 2024-03-18T01:10:53.197 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:10:52.554+0000 7fcd611e3700 1 -- 172.21.15.29:0/2546815462 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7fcd5800be50 con 0x7fcd5c076bc0 2024-03-18T01:10:53.197 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:10:52.554+0000 7fcd611e3700 1 -- 172.21.15.29:0/2546815462 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcd58010380 con 0x7fcd5c076bc0 2024-03-18T01:10:53.197 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:10:52.555+0000 7fcd64449700 1 -- 172.21.15.29:0/2546815462 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcd5c076bc0 msgr2=0x7fcd5c105310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:10:53.198 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:10:52.555+0000 7fcd64449700 1 --2- 172.21.15.29:0/2546815462 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcd5c076bc0 0x7fcd5c105310 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fcd580098e0 tx=0x7fcd58009250 comp rx=0 tx=0).stop 2024-03-18T01:10:53.198 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:10:52.555+0000 7fcd64449700 1 -- 172.21.15.29:0/2546815462 shutdown_connections 2024-03-18T01:10:53.198 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:10:52.555+0000 7fcd64449700 1 --2- 172.21.15.29:0/2546815462 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcd5c076bc0 0x7fcd5c105310 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:10:53.198 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:10:52.555+0000 7fcd64449700 1 -- 172.21.15.29:0/2546815462 >> 172.21.15.29:0/2546815462 conn(0x7fcd5c0fcd80 msgr2=0x7fcd5c0ff190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:10:53.198 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:10:52.555+0000 7fcd64449700 1 -- 172.21.15.29:0/2546815462 shutdown_connections 2024-03-18T01:10:53.198 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:10:52.555+0000 7fcd64449700 1 -- 172.21.15.29:0/2546815462 wait complete. 2024-03-18T01:10:53.198 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:10:52.556+0000 7fcd64449700 1 Processor -- start 2024-03-18T01:10:53.198 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:10:52.556+0000 7fcd64449700 1 -- start start 2024-03-18T01:10:53.198 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:10:52.556+0000 7fcd64449700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcd5c076bc0 0x7fcd5c111540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:10:53.198 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:10:52.556+0000 7fcd64449700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd5c111a80 con 0x7fcd5c076bc0 2024-03-18T01:10:53.198 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:10:52.556+0000 7fcd621e5700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcd5c076bc0 0x7fcd5c111540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:10:53.198 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:10:52.556+0000 7fcd621e5700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcd5c076bc0 0x7fcd5c111540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51832/0 (socket says 172.21.15.29:51832) 2024-03-18T01:10:53.199 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:10:52.556+0000 7fcd621e5700 1 -- 172.21.15.29:0/958991352 learned_addr learned my addr 172.21.15.29:0/958991352 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-18T01:10:53.199 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:10:52.556+0000 7fcd621e5700 1 -- 172.21.15.29:0/958991352 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd580094f0 con 0x7fcd5c076bc0 2024-03-18T01:10:53.199 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:10:52.556+0000 7fcd621e5700 1 --2- 172.21.15.29:0/958991352 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcd5c076bc0 0x7fcd5c111540 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fcd58006b00 tx=0x7fcd58009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:10:53.199 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:10:52.556+0000 7fcd52ffd700 1 -- 172.21.15.29:0/958991352 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcd580107c0 con 0x7fcd5c076bc0 2024-03-18T01:10:53.199 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:10:52.556+0000 7fcd52ffd700 1 -- 172.21.15.29:0/958991352 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7fcd58010db0 con 0x7fcd5c076bc0 2024-03-18T01:10:53.199 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:10:52.557+0000 7fcd64449700 1 -- 172.21.15.29:0/958991352 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd5c111ce0 con 0x7fcd5c076bc0 2024-03-18T01:10:53.199 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:10:52.557+0000 7fcd52ffd700 1 -- 172.21.15.29:0/958991352 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcd5801acf0 con 0x7fcd5c076bc0 2024-03-18T01:10:53.199 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:10:52.557+0000 7fcd64449700 1 -- 172.21.15.29:0/958991352 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd5c1127f0 con 0x7fcd5c076bc0 2024-03-18T01:10:53.199 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:10:52.557+0000 7fcd52ffd700 1 -- 172.21.15.29:0/958991352 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7fcd58024440 con 0x7fcd5c076bc0 2024-03-18T01:10:53.199 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:10:52.557+0000 7fcd52ffd700 1 -- 172.21.15.29:0/958991352 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7fcd58019710 con 0x7fcd5c076bc0 2024-03-18T01:10:53.199 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:10:52.558+0000 7fcd64449700 1 -- 172.21.15.29:0/958991352 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd5c05f5f0 con 0x7fcd5c076bc0 2024-03-18T01:10:53.199 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:10:52.560+0000 7fcd52ffd700 1 -- 172.21.15.29:0/958991352 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7fcd5801a530 con 0x7fcd5c076bc0 2024-03-18T01:10:53.199 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:10:52.630+0000 7fcd64449700 1 -- 172.21.15.29:0/958991352 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7fcd5c1066e0 con 0x7fcd5c076bc0 2024-03-18T01:10:53.200 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:10:52.630+0000 7fcd52ffd700 1 -- 172.21.15.29:0/958991352 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) v1 ==== 76+0+175 (secure 0 0 0) 0x7fcd58031030 con 0x7fcd5c076bc0 2024-03-18T01:10:53.200 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:10:52.631+0000 7fcd64449700 1 -- 172.21.15.29:0/958991352 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcd5c076bc0 msgr2=0x7fcd5c111540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:10:53.200 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:10:52.631+0000 7fcd64449700 1 --2- 172.21.15.29:0/958991352 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcd5c076bc0 0x7fcd5c111540 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fcd58006b00 tx=0x7fcd58009250 comp rx=0 tx=0).stop 2024-03-18T01:10:53.200 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:10:52.631+0000 7fcd64449700 1 -- 172.21.15.29:0/958991352 shutdown_connections 2024-03-18T01:10:53.200 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:10:52.631+0000 7fcd64449700 1 --2- 172.21.15.29:0/958991352 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcd5c076bc0 0x7fcd5c111540 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:10:53.200 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:10:52.631+0000 7fcd64449700 1 -- 172.21.15.29:0/958991352 >> 172.21.15.29:0/958991352 conn(0x7fcd5c0fcd80 msgr2=0x7fcd5c0fd950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:10:53.200 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:10:52.631+0000 7fcd64449700 1 -- 172.21.15.29:0/958991352 shutdown_connections 2024-03-18T01:10:53.200 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:10:52.631+0000 7fcd64449700 1 -- 172.21.15.29:0/958991352 wait complete. 2024-03-18T01:10:53.200 INFO:teuthology.orchestra.run.smithi029.stdout:Restarting the monitor... 2024-03-18T01:10:53.459 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:10:53 smithi029 systemd[1]: Stopping Ceph mon.smithi029 for 37f51182-e4c4-11ee-95c9-87774f69a715... 2024-03-18T01:10:53.819 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:10:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[28960]: 2024-03-18T01:10:53.459+0000 7f637c0a9700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi029 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T01:10:53.819 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:10:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[28960]: 2024-03-18T01:10:53.459+0000 7f637c0a9700 -1 mon.smithi029@0(leader) e1 *** Got Signal Terminated *** 2024-03-18T01:10:53.819 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:10:53 smithi029 podman[29208]: 2024-03-18 01:10:53.704991987 +0000 UTC m=+0.349341655 container died 55d7394db9ec6a91d24d4dd09aa3f7a4106d02709dcb63312cf15f737cc1f547 (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=quincy-6716c9f, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-18T01:10:54.319 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:10:54 smithi029 podman[29208]: 2024-03-18 01:10:54.045894912 +0000 UTC m=+0.690244584 container cleanup 55d7394db9ec6a91d24d4dd09aa3f7a4106d02709dcb63312cf15f737cc1f547 (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, RELEASE=quincy-6716c9f, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-18T01:10:54.319 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:10:54 smithi029 bash[29208]: ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029 2024-03-18T01:10:54.319 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:10:54 smithi029 podman[29222]: 2024-03-18 01:10:54.198295713 +0000 UTC m=+0.496857494 container remove 55d7394db9ec6a91d24d4dd09aa3f7a4106d02709dcb63312cf15f737cc1f547 (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, RELEASE=quincy-6716c9f, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5) 2024-03-18T01:10:54.766 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:10:54 smithi029 systemd[1]: ceph-37f51182-e4c4-11ee-95c9-87774f69a715@mon.smithi029.service: Deactivated successfully. 2024-03-18T01:10:54.766 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:10:54 smithi029 systemd[1]: Stopped Ceph mon.smithi029 for 37f51182-e4c4-11ee-95c9-87774f69a715. 2024-03-18T01:10:54.766 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:10:54 smithi029 systemd[1]: Starting Ceph mon.smithi029 for 37f51182-e4c4-11ee-95c9-87774f69a715... 2024-03-18T01:10:54.766 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:10:54 smithi029 podman[29312]: 2024-03-18 01:10:54.53226543 +0000 UTC m=+0.018212927 image pull 19857db3290c58169e44a8aec8ef9646981a77ffd5b146ecef1b6faf59d6c3ab quay.ceph.io/ceph-ci/ceph:quincy 2024-03-18T01:10:54.767 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:10:54 smithi029 podman[29312]: 2024-03-18 01:10:54.766700846 +0000 UTC m=+0.252648382 container create 0a7fff1d6dfcd675abb5a89188965e3ff6387fe9e1977d5eb7eafe0a990b5420 (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=quincy-6716c9f, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.33.5) 2024-03-18T01:10:55.016 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:10:54 smithi029 podman[29312]: 2024-03-18 01:10:54.98762658 +0000 UTC m=+0.473574085 container init 0a7fff1d6dfcd675abb5a89188965e3ff6387fe9e1977d5eb7eafe0a990b5420 (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=quincy-6716c9f, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-03-18T01:10:55.016 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:10:54 smithi029 podman[29312]: 2024-03-18 01:10:54.990436854 +0000 UTC m=+0.476384346 container start 0a7fff1d6dfcd675abb5a89188965e3ff6387fe9e1977d5eb7eafe0a990b5420 (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=quincy-6716c9f, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-03-18T01:10:55.017 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:10:55 smithi029 ceph-mon[29331]: set uid:gid to 167:167 (ceph:ceph) 2024-03-18T01:10:55.017 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:10:55 smithi029 ceph-mon[29331]: ceph version 17.2.7-684-g6716c9f3 (6716c9f3976cf9fc2fd546e9f1182c2d4421b451) quincy (stable), process ceph-mon, pid 2 2024-03-18T01:10:55.017 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:10:55 smithi029 ceph-mon[29331]: pidfile_write: ignore empty --pid-file 2024-03-18T01:10:55.017 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:10:55 smithi029 ceph-mon[29331]: load: jerasure load: lrc 2024-03-18T01:10:55.017 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:10:55 smithi029 ceph-mon[29331]: rocksdb: RocksDB version: 6.15.5 2024-03-18T01:10:55.017 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:10:55 smithi029 ceph-mon[29331]: rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-03-18T01:10:55.017 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:10:55 smithi029 ceph-mon[29331]: rocksdb: Compile date Mar 16 2024 2024-03-18T01:10:55.017 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:10:55 smithi029 ceph-mon[29331]: rocksdb: DB SUMMARY 2024-03-18T01:10:55.017 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:10:55 smithi029 ceph-mon[29331]: rocksdb: DB Session ID: IOYGQ94IS8HFO8YX3Z9N 2024-03-18T01:10:55.017 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:10:55 smithi029 ceph-mon[29331]: rocksdb: CURRENT file: CURRENT 2024-03-18T01:10:55.017 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:10:55 smithi029 ceph-mon[29331]: rocksdb: IDENTITY file: IDENTITY 2024-03-18T01:10:55.018 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:10:55 smithi029 ceph-mon[29331]: rocksdb: MANIFEST file: MANIFEST-000009 size: 131 Bytes 2024-03-18T01:10:55.018 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:10:55 smithi029 ceph-mon[29331]: rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi029/store.db dir, Total Num: 1, files: 000008.sst 2024-03-18T01:10:55.018 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:10:55 smithi029 ceph-mon[29331]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi029/store.db: 000010.log size: 97278 ; 2024-03-18T01:10:55.018 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:10:55 smithi029 ceph-mon[29331]: rocksdb: Options.error_if_exists: 0 2024-03-18T01:10:55.018 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:10:55 smithi029 ceph-mon[29331]: rocksdb: Options.create_if_missing: 0 2024-03-18T01:10:55.018 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:10:55 smithi029 ceph-mon[29331]: rocksdb: Options.paranoid_checks: 1 2024-03-18T01:10:55.018 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:10:55 smithi029 ceph-mon[29331]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-18T01:10:55.018 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:10:55 smithi029 ceph-mon[29331]: rocksdb: Options.env: 0x5567ee32ea80 2024-03-18T01:10:55.018 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:10:55 smithi029 ceph-mon[29331]: rocksdb: Options.fs: Posix File System 2024-03-18T01:10:55.018 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:10:55 smithi029 ceph-mon[29331]: rocksdb: Options.info_log: 0x5567f08cdc00 2024-03-18T01:10:55.018 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:10:55 smithi029 ceph-mon[29331]: rocksdb: Options.max_file_opening_threads: 16 2024-03-18T01:10:55.019 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:10:55 smithi029 ceph-mon[29331]: rocksdb: Options.statistics: (nil) 2024-03-18T01:10:55.019 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:10:55 smithi029 ceph-mon[29331]: rocksdb: Options.use_fsync: 0 2024-03-18T01:10:55.019 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:10:55 smithi029 ceph-mon[29331]: rocksdb: Options.max_log_file_size: 0 2024-03-18T01:10:55.020 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:10:55 smithi029 ceph-mon[29331]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-18T01:10:55.020 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:10:55 smithi029 ceph-mon[29331]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-18T01:10:55.020 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:10:55 smithi029 ceph-mon[29331]: rocksdb: Options.keep_log_file_num: 1000 2024-03-18T01:10:55.021 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:10:55 smithi029 ceph-mon[29331]: rocksdb: Options.recycle_log_file_num: 0 2024-03-18T01:10:55.021 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:10:55 smithi029 ceph-mon[29331]: rocksdb: Options.allow_fallocate: 1 2024-03-18T01:10:55.021 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:10:55 smithi029 ceph-mon[29331]: rocksdb: Options.allow_mmap_reads: 0 2024-03-18T01:10:55.021 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:10:55 smithi029 ceph-mon[29331]: rocksdb: Options.allow_mmap_writes: 0 2024-03-18T01:10:55.021 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:10:55 smithi029 ceph-mon[29331]: rocksdb: Options.use_direct_reads: 0 2024-03-18T01:10:55.021 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:10:55 smithi029 ceph-mon[29331]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-18T01:10:55.021 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:10:55 smithi029 ceph-mon[29331]: rocksdb: Options.create_missing_column_families: 0 2024-03-18T01:10:55.021 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:10:55 smithi029 ceph-mon[29331]: rocksdb: Options.db_log_dir: 2024-03-18T01:10:55.021 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:10:55 smithi029 ceph-mon[29331]: rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-smithi029/store.db 2024-03-18T01:10:55.021 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:10:55 smithi029 ceph-mon[29331]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-18T01:10:55.021 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:10:55 smithi029 ceph-mon[29331]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-18T01:10:55.021 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:10:55 smithi029 ceph-mon[29331]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-18T01:10:55.022 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:10:55 smithi029 ceph-mon[29331]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-18T01:10:55.022 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:10:55 smithi029 ceph-mon[29331]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-18T01:10:55.022 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:10:55 smithi029 ceph-mon[29331]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-18T01:10:55.022 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:10:55 smithi029 ceph-mon[29331]: rocksdb: Options.advise_random_on_open: 1 2024-03-18T01:10:55.022 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:10:55 smithi029 ceph-mon[29331]: rocksdb: Options.db_write_buffer_size: 0 2024-03-18T01:10:55.022 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:10:55 smithi029 ceph-mon[29331]: rocksdb: Options.write_buffer_manager: 0x5567f1642ab0 2024-03-18T01:10:55.022 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:10:55 smithi029 ceph-mon[29331]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-18T01:10:55.022 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:10:55 smithi029 ceph-mon[29331]: rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-03-18T01:10:55.022 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:10:55 smithi029 ceph-mon[29331]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-18T01:10:55.022 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:10:55 smithi029 ceph-mon[29331]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-18T01:10:55.022 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:10:55 smithi029 ceph-mon[29331]: rocksdb: Options.rate_limiter: (nil) 2024-03-18T01:10:55.023 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:10:55 smithi029 ceph-mon[29331]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-18T01:10:55.023 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:10:55 smithi029 ceph-mon[29331]: rocksdb: Options.wal_recovery_mode: 2 2024-03-18T01:10:55.023 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:10:55 smithi029 ceph-mon[29331]: rocksdb: Options.enable_thread_tracking: 0 2024-03-18T01:10:55.023 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:10:55 smithi029 ceph-mon[29331]: rocksdb: Options.enable_pipelined_write: 0 2024-03-18T01:10:55.023 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:10:55 smithi029 ceph-mon[29331]: rocksdb: Options.unordered_write: 0 2024-03-18T01:10:55.023 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:10:55 smithi029 ceph-mon[29331]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-18T01:10:55.023 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:10:55 smithi029 ceph-mon[29331]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-18T01:10:55.023 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:10:55 smithi029 ceph-mon[29331]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-18T01:10:55.023 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:10:55 smithi029 ceph-mon[29331]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-18T01:10:55.023 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:10:55 smithi029 ceph-mon[29331]: rocksdb: Options.row_cache: None 2024-03-18T01:10:55.024 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:10:55 smithi029 ceph-mon[29331]: rocksdb: Options.wal_filter: None 2024-03-18T01:10:55.024 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:10:55 smithi029 ceph-mon[29331]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-18T01:10:55.024 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:10:55 smithi029 ceph-mon[29331]: rocksdb: Options.allow_ingest_behind: 0 2024-03-18T01:10:55.024 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:10:55 smithi029 ceph-mon[29331]: rocksdb: Options.preserve_deletes: 0 2024-03-18T01:10:55.024 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:10:55 smithi029 ceph-mon[29331]: rocksdb: Options.two_write_queues: 0 2024-03-18T01:10:55.024 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:10:55 smithi029 ceph-mon[29331]: rocksdb: Options.manual_wal_flush: 0 2024-03-18T01:10:55.024 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:10:55 smithi029 ceph-mon[29331]: rocksdb: Options.atomic_flush: 0 2024-03-18T01:10:55.024 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:10:55 smithi029 ceph-mon[29331]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-18T01:10:55.024 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:10:55 smithi029 ceph-mon[29331]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-18T01:10:55.024 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:10:55 smithi029 ceph-mon[29331]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-18T01:10:55.024 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:10:55 smithi029 ceph-mon[29331]: rocksdb: Options.log_readahead_size: 0 2024-03-18T01:10:55.025 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:10:55 smithi029 ceph-mon[29331]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-18T01:10:55.025 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:10:55 smithi029 ceph-mon[29331]: rocksdb: Options.best_efforts_recovery: 0 2024-03-18T01:10:55.025 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:10:55 smithi029 ceph-mon[29331]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-18T01:10:55.025 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:10:55 smithi029 ceph-mon[29331]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-18T01:10:55.025 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:10:55 smithi029 ceph-mon[29331]: rocksdb: Options.allow_data_in_errors: 0 2024-03-18T01:10:55.025 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:10:55 smithi029 ceph-mon[29331]: rocksdb: Options.db_host_id: __hostname__ 2024-03-18T01:10:55.025 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:10:55 smithi029 ceph-mon[29331]: rocksdb: Options.max_background_jobs: 2 2024-03-18T01:10:55.025 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:10:55 smithi029 ceph-mon[29331]: rocksdb: Options.max_background_compactions: -1 2024-03-18T01:10:55.025 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:10:55 smithi029 ceph-mon[29331]: rocksdb: Options.max_subcompactions: 1 2024-03-18T01:10:55.025 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:10:55 smithi029 ceph-mon[29331]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-18T01:10:55.025 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:10:55 smithi029 ceph-mon[29331]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-18T01:10:55.025 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:10:55 smithi029 ceph-mon[29331]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-18T01:10:55.025 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:10:55 smithi029 ceph-mon[29331]: rocksdb: Options.max_total_wal_size: 0 2024-03-18T01:10:55.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:10:55 smithi029 ceph-mon[29331]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-18T01:10:55.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:10:55 smithi029 ceph-mon[29331]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-18T01:10:55.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:10:55 smithi029 ceph-mon[29331]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-18T01:10:55.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:10:55 smithi029 ceph-mon[29331]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-18T01:10:55.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:10:55 smithi029 ceph-mon[29331]: rocksdb: Options.max_open_files: -1 2024-03-18T01:10:55.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:10:55 smithi029 ceph-mon[29331]: rocksdb: Options.bytes_per_sync: 0 2024-03-18T01:10:55.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:10:55 smithi029 ceph-mon[29331]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-18T01:10:55.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:10:55 smithi029 ceph-mon[29331]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-18T01:10:55.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:10:55 smithi029 ceph-mon[29331]: rocksdb: Options.compaction_readahead_size: 0 2024-03-18T01:10:55.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:10:55 smithi029 ceph-mon[29331]: rocksdb: Options.max_background_flushes: -1 2024-03-18T01:10:55.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:10:55 smithi029 ceph-mon[29331]: rocksdb: Compression algorithms supported: 2024-03-18T01:10:55.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:10:55 smithi029 ceph-mon[29331]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-18T01:10:55.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:10:55 smithi029 ceph-mon[29331]: rocksdb: kZSTD supported: 0 2024-03-18T01:10:55.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:10:55 smithi029 ceph-mon[29331]: rocksdb: kXpressCompression supported: 0 2024-03-18T01:10:55.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:10:55 smithi029 ceph-mon[29331]: rocksdb: kLZ4HCCompression supported: 1 2024-03-18T01:10:55.027 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:10:55 smithi029 ceph-mon[29331]: rocksdb: kZlibCompression supported: 1 2024-03-18T01:10:55.027 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:10:55 smithi029 ceph-mon[29331]: rocksdb: kSnappyCompression supported: 1 2024-03-18T01:10:55.027 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:10:55 smithi029 ceph-mon[29331]: rocksdb: kLZ4Compression supported: 1 2024-03-18T01:10:55.027 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:10:55 smithi029 ceph-mon[29331]: rocksdb: kBZip2Compression supported: 0 2024-03-18T01:10:55.027 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:10:55 smithi029 ceph-mon[29331]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-18T01:10:55.027 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:10:55 smithi029 ceph-mon[29331]: rocksdb: [db/version_set.cc:4724] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi029/store.db/MANIFEST-000009 2024-03-18T01:10:55.027 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:10:55 smithi029 ceph-mon[29331]: rocksdb: [db/column_family.cc:595] --------------- Options for column family [default]: 2024-03-18T01:10:55.027 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:10:55 smithi029 ceph-mon[29331]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-18T01:10:55.027 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:10:55 smithi029 ceph-mon[29331]: rocksdb: Options.merge_operator: 2024-03-18T01:10:55.027 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:10:55 smithi029 ceph-mon[29331]: rocksdb: Options.compaction_filter: None 2024-03-18T01:10:55.027 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:10:55 smithi029 ceph-mon[29331]: rocksdb: Options.compaction_filter_factory: None 2024-03-18T01:10:55.027 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:10:55 smithi029 ceph-mon[29331]: rocksdb: Options.sst_partitioner_factory: None 2024-03-18T01:10:55.027 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:10:55 smithi029 ceph-mon[29331]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-18T01:10:55.027 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:10:55 smithi029 ceph-mon[29331]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-18T01:10:55.027 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:10:55 smithi029 ceph-mon[29331]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5567f0922f58) 2024-03-18T01:10:55.028 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: cache_index_and_filter_blocks: 1 2024-03-18T01:10:55.028 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-18T01:10:55.028 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-18T01:10:55.028 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: pin_top_level_index_and_filter: 1 2024-03-18T01:10:55.028 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: index_type: 0 2024-03-18T01:10:55.028 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: data_block_index_type: 0 2024-03-18T01:10:55.028 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: index_shortening: 1 2024-03-18T01:10:55.028 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-18T01:10:55.028 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: hash_index_allow_collision: 1 2024-03-18T01:10:55.028 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: checksum: 1 2024-03-18T01:10:55.028 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: no_block_cache: 0 2024-03-18T01:10:55.028 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: block_cache: 0x5567f0955090 2024-03-18T01:10:55.028 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: block_cache_name: BinnedLRUCache 2024-03-18T01:10:55.028 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: block_cache_options: 2024-03-18T01:10:55.029 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: capacity : 536870912 2024-03-18T01:10:55.029 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: num_shard_bits : 4 2024-03-18T01:10:55.029 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: strict_capacity_limit : 0 2024-03-18T01:10:55.029 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: high_pri_pool_ratio: 0.000 2024-03-18T01:10:55.029 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: block_cache_compressed: (nil) 2024-03-18T01:10:55.029 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: persistent_cache: (nil) 2024-03-18T01:10:55.029 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: block_size: 4096 2024-03-18T01:10:55.029 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: block_size_deviation: 10 2024-03-18T01:10:55.029 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: block_restart_interval: 16 2024-03-18T01:10:55.029 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: index_block_restart_interval: 1 2024-03-18T01:10:55.029 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: metadata_block_size: 4096 2024-03-18T01:10:55.029 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: partition_filters: 0 2024-03-18T01:10:55.029 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: use_delta_encoding: 1 2024-03-18T01:10:55.029 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: filter_policy: rocksdb.BuiltinBloomFilter 2024-03-18T01:10:55.029 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: whole_key_filtering: 1 2024-03-18T01:10:55.029 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: verify_compression: 0 2024-03-18T01:10:55.030 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: read_amp_bytes_per_bit: 0 2024-03-18T01:10:55.030 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: format_version: 4 2024-03-18T01:10:55.030 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: enable_index_compression: 1 2024-03-18T01:10:55.030 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: block_align: 0 2024-03-18T01:10:55.030 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:10:55 smithi029 ceph-mon[29331]: rocksdb: Options.write_buffer_size: 33554432 2024-03-18T01:10:55.030 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:10:55 smithi029 ceph-mon[29331]: rocksdb: Options.max_write_buffer_number: 2 2024-03-18T01:10:55.030 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:10:55 smithi029 ceph-mon[29331]: rocksdb: Options.compression: NoCompression 2024-03-18T01:10:55.030 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:10:55 smithi029 ceph-mon[29331]: rocksdb: Options.bottommost_compression: Disabled 2024-03-18T01:10:55.030 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:10:55 smithi029 ceph-mon[29331]: rocksdb: Options.prefix_extractor: nullptr 2024-03-18T01:10:55.030 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:10:55 smithi029 ceph-mon[29331]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-18T01:10:55.030 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:10:55 smithi029 ceph-mon[29331]: rocksdb: Options.num_levels: 7 2024-03-18T01:10:55.031 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:10:55 smithi029 ceph-mon[29331]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-18T01:10:55.031 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:10:55 smithi029 ceph-mon[29331]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-18T01:10:55.031 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:10:55 smithi029 ceph-mon[29331]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-18T01:10:55.031 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:10:55 smithi029 ceph-mon[29331]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-18T01:10:55.031 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:10:55 smithi029 ceph-mon[29331]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-18T01:10:55.031 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:10:55 smithi029 ceph-mon[29331]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-18T01:10:55.031 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:10:55 smithi029 ceph-mon[29331]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-18T01:10:55.031 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:10:55 smithi029 ceph-mon[29331]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-18T01:10:55.031 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:10:55 smithi029 ceph-mon[29331]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-18T01:10:55.031 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:10:55 smithi029 ceph-mon[29331]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-18T01:10:55.031 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:10:55 smithi029 ceph-mon[29331]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-18T01:10:55.031 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:10:55 smithi029 ceph-mon[29331]: rocksdb: Options.compression_opts.level: 32767 2024-03-18T01:10:55.031 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:10:55 smithi029 ceph-mon[29331]: rocksdb: Options.compression_opts.strategy: 0 2024-03-18T01:10:55.031 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:10:55 smithi029 ceph-mon[29331]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-18T01:10:55.031 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:10:55 smithi029 ceph-mon[29331]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-18T01:10:55.031 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:10:55 smithi029 ceph-mon[29331]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-18T01:10:55.032 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:10:55 smithi029 ceph-mon[29331]: rocksdb: Options.compression_opts.enabled: false 2024-03-18T01:10:55.032 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:10:55 smithi029 ceph-mon[29331]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-18T01:10:55.032 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:10:55 smithi029 ceph-mon[29331]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-18T01:10:55.032 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:10:55 smithi029 ceph-mon[29331]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-18T01:10:55.032 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:10:55 smithi029 ceph-mon[29331]: rocksdb: Options.target_file_size_base: 67108864 2024-03-18T01:10:55.032 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:10:55 smithi029 ceph-mon[29331]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-18T01:10:55.032 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:10:55 smithi029 ceph-mon[29331]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-18T01:10:55.032 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:10:55 smithi029 ceph-mon[29331]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-18T01:10:55.032 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:10:55 smithi029 ceph-mon[29331]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-18T01:10:55.032 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:10:55 smithi029 ceph-mon[29331]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-18T01:10:55.032 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:10:55 smithi029 ceph-mon[29331]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-18T01:10:55.032 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:10:55 smithi029 ceph-mon[29331]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-18T01:10:55.032 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:10:55 smithi029 ceph-mon[29331]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-18T01:10:55.032 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:10:55 smithi029 ceph-mon[29331]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-18T01:10:55.032 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:10:55 smithi029 ceph-mon[29331]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-18T01:10:55.032 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:10:55 smithi029 ceph-mon[29331]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-18T01:10:55.033 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:10:55 smithi029 ceph-mon[29331]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-18T01:10:55.033 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:10:55 smithi029 ceph-mon[29331]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-18T01:10:55.033 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:10:55 smithi029 ceph-mon[29331]: rocksdb: Options.arena_block_size: 4194304 2024-03-18T01:10:55.033 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:10:55 smithi029 ceph-mon[29331]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-18T01:10:55.033 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:10:55 smithi029 ceph-mon[29331]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-18T01:10:55.033 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:10:55 smithi029 ceph-mon[29331]: rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-03-18T01:10:55.033 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:10:55 smithi029 ceph-mon[29331]: rocksdb: Options.disable_auto_compactions: 0 2024-03-18T01:10:55.033 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:10:55 smithi029 ceph-mon[29331]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-18T01:10:55.033 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:10:55 smithi029 ceph-mon[29331]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-18T01:10:55.033 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:10:55 smithi029 ceph-mon[29331]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-18T01:10:55.033 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:10:55 smithi029 ceph-mon[29331]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-18T01:10:55.033 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:10:55 smithi029 ceph-mon[29331]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-18T01:10:55.033 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:10:55 smithi029 ceph-mon[29331]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-18T01:10:55.033 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:10:55 smithi029 ceph-mon[29331]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-18T01:10:55.033 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:10:55 smithi029 ceph-mon[29331]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-18T01:10:55.034 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:10:55 smithi029 ceph-mon[29331]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-18T01:10:55.034 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:10:55 smithi029 ceph-mon[29331]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-18T01:10:55.034 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:10:55 smithi029 ceph-mon[29331]: rocksdb: Options.table_properties_collectors: 2024-03-18T01:10:55.034 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:10:55 smithi029 ceph-mon[29331]: rocksdb: Options.inplace_update_support: 0 2024-03-18T01:10:55.034 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:10:55 smithi029 ceph-mon[29331]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-18T01:10:55.034 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:10:55 smithi029 ceph-mon[29331]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-18T01:10:55.034 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:10:55 smithi029 ceph-mon[29331]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-18T01:10:55.034 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:10:55 smithi029 ceph-mon[29331]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-18T01:10:55.034 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:10:55 smithi029 ceph-mon[29331]: rocksdb: Options.bloom_locality: 0 2024-03-18T01:10:55.034 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:10:55 smithi029 ceph-mon[29331]: rocksdb: Options.max_successive_merges: 0 2024-03-18T01:10:55.034 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:10:55 smithi029 ceph-mon[29331]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-18T01:10:55.034 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:10:55 smithi029 ceph-mon[29331]: rocksdb: Options.paranoid_file_checks: 0 2024-03-18T01:10:55.034 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:10:55 smithi029 ceph-mon[29331]: rocksdb: Options.force_consistency_checks: 1 2024-03-18T01:10:55.035 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:10:55 smithi029 ceph-mon[29331]: rocksdb: Options.report_bg_io_stats: 0 2024-03-18T01:10:55.035 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:10:55 smithi029 ceph-mon[29331]: rocksdb: Options.ttl: 2592000 2024-03-18T01:10:55.035 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:10:55 smithi029 ceph-mon[29331]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-18T01:10:55.035 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:10:55 smithi029 ceph-mon[29331]: rocksdb: Options.enable_blob_files: false 2024-03-18T01:10:55.035 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:10:55 smithi029 ceph-mon[29331]: rocksdb: Options.min_blob_size: 0 2024-03-18T01:10:55.035 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:10:55 smithi029 ceph-mon[29331]: rocksdb: Options.blob_file_size: 268435456 2024-03-18T01:10:55.035 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:10:55 smithi029 ceph-mon[29331]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-18T01:10:55.035 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:10:55 smithi029 ceph-mon[29331]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-18T01:10:55.035 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:10:55 smithi029 ceph-mon[29331]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-18T01:10:55.035 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:10:55 smithi029 ceph-mon[29331]: rocksdb: [db/version_set.cc:4764] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi029/store.db/MANIFEST-000009 succeeded,manifest_file_number is 9, next_file_number is 11, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-03-18T01:10:55.035 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:10:55 smithi029 ceph-mon[29331]: rocksdb: [db/version_set.cc:4779] Column family [default] (ID 0), log number is 5 2024-03-18T01:10:55.035 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:10:55 smithi029 ceph-mon[29331]: rocksdb: [db/version_set.cc:4082] Creating manifest 13 2024-03-18T01:10:55.035 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:10:55 smithi029 ceph-mon[29331]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710724255015852, "job": 1, "event": "recovery_started", "wal_files": [10]} 2024-03-18T01:10:55.036 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:10:55 smithi029 ceph-mon[29331]: rocksdb: [db/db_impl/db_impl_open.cc:845] Recovering log #10 mode 2 2024-03-18T01:10:55.036 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:10:55 smithi029 ceph-mon[29331]: rocksdb: [table/block_based/filter_policy.cc:991] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2024-03-18T01:10:55.036 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:10:55 smithi029 ceph-mon[29331]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710724255016607, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 14, "file_size": 92550, "file_checksum": "", "file_checksum_func_name": "Unknown", "table_properties": {"data_size": 90645, "index_size": 265, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 709, "raw_key_size": 13356, "raw_average_key_size": 49, "raw_value_size": 83360, "raw_average_value_size": 309, "num_data_blocks": 11, "num_entries": 269, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1710724255, "oldest_key_time": 0, "file_creation_time": 0, "db_id": "d403f7d5-a983-4382-87f9-9755adb1a7d4", "db_session_id": "IOYGQ94IS8HFO8YX3Z9N"}} 2024-03-18T01:10:55.036 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:10:55 smithi029 ceph-mon[29331]: rocksdb: [db/version_set.cc:4082] Creating manifest 15 2024-03-18T01:10:55.036 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:10:55 smithi029 ceph-mon[29331]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710724255016987, "job": 1, "event": "recovery_finished"} 2024-03-18T01:10:55.036 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:10:55 smithi029 ceph-mon[29331]: rocksdb: [file/delete_scheduler.cc:69] Deleted file /var/lib/ceph/mon/ceph-smithi029/store.db/000010.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-18T01:10:55.036 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:10:55 smithi029 ceph-mon[29331]: rocksdb: [db/db_impl/db_impl_open.cc:1700] SstFileManager instance 0x5567f08e9180 2024-03-18T01:10:55.036 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:10:55 smithi029 ceph-mon[29331]: rocksdb: DB pointer 0x5567f09be000 2024-03-18T01:10:55.036 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:10:55 smithi029 ceph-mon[29331]: rocksdb: [db/db_impl/db_impl.cc:901] ------- DUMPING STATS ------- 2024-03-18T01:10:55.036 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:10:55 smithi029 ceph-mon[29331]: rocksdb: [db/db_impl/db_impl.cc:903] 2024-03-18T01:10:55.036 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: ** DB Stats ** 2024-03-18T01:10:55.036 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-18T01:10:55.036 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-18T01:10:55.036 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-18T01:10:55.036 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-18T01:10:55.037 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-18T01:10:55.037 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-03-18T01:10:55.037 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-18T01:10:55.037 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: 2024-03-18T01:10:55.037 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: ** Compaction Stats [default] ** 2024-03-18T01:10:55.037 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-03-18T01:10:55.037 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-18T01:10:55.037 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: L0 2/0 92.18 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 169.7 0.00 0.00 1 0.001 0 0 2024-03-18T01:10:55.037 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: Sum 2/0 92.18 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 169.7 0.00 0.00 1 0.001 0 0 2024-03-18T01:10:55.037 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 169.7 0.00 0.00 1 0.001 0 0 2024-03-18T01:10:55.037 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: 2024-03-18T01:10:55.037 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: ** Compaction Stats [default] ** 2024-03-18T01:10:55.037 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-03-18T01:10:55.038 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-18T01:10:55.038 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 169.7 0.00 0.00 1 0.001 0 0 2024-03-18T01:10:55.038 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-18T01:10:55.038 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-18T01:10:55.038 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-18T01:10:55.038 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-18T01:10:55.038 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-18T01:10:55.038 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-18T01:10:55.038 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: Cumulative compaction: 0.00 GB write, 28.83 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-18T01:10:55.038 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: Interval compaction: 0.00 GB write, 28.83 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-18T01:10:55.038 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-18T01:10:55.038 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: 2024-03-18T01:10:55.039 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-18T01:10:55.039 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: 2024-03-18T01:10:55.039 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: ** Compaction Stats [default] ** 2024-03-18T01:10:55.039 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-03-18T01:10:55.039 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-18T01:10:55.039 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: L0 2/0 92.18 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 169.7 0.00 0.00 1 0.001 0 0 2024-03-18T01:10:55.039 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: Sum 2/0 92.18 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 169.7 0.00 0.00 1 0.001 0 0 2024-03-18T01:10:55.039 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-03-18T01:10:55.039 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: 2024-03-18T01:10:55.039 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: ** Compaction Stats [default] ** 2024-03-18T01:10:55.039 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-03-18T01:10:55.039 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-18T01:10:55.039 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 169.7 0.00 0.00 1 0.001 0 0 2024-03-18T01:10:55.039 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-18T01:10:55.039 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-18T01:10:55.039 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-18T01:10:55.040 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-18T01:10:55.040 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-18T01:10:55.040 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-18T01:10:55.040 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: Cumulative compaction: 0.00 GB write, 28.55 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-18T01:10:55.040 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-18T01:10:55.040 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-18T01:10:55.040 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: 2024-03-18T01:10:55.040 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-18T01:10:55.040 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:10:55 smithi029 ceph-mon[29331]: starting mon.smithi029 rank 0 at public addrs [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] at bind addrs [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi029 fsid 37f51182-e4c4-11ee-95c9-87774f69a715 2024-03-18T01:10:55.040 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:10:55 smithi029 ceph-mon[29331]: mon.smithi029@-1(???) e1 preinit fsid 37f51182-e4c4-11ee-95c9-87774f69a715 2024-03-18T01:10:55.040 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:10:55 smithi029 ceph-mon[29331]: mon.smithi029@-1(???).mds e1 new map 2024-03-18T01:10:55.040 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:10:55 smithi029 ceph-mon[29331]: mon.smithi029@-1(???).mds e1 print_map 2024-03-18T01:10:55.040 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: e1 2024-03-18T01:10:55.040 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-18T01:10:55.040 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-18T01:10:55.041 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: legacy client fscid: -1 2024-03-18T01:10:55.041 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: 2024-03-18T01:10:55.041 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: No filesystems configured 2024-03-18T01:10:55.143 INFO:teuthology.orchestra.run.smithi029.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-03-18T01:10:55.267 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:10:55 smithi029 ceph-mon[29331]: mon.smithi029@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-18T01:10:55.267 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:10:55 smithi029 ceph-mon[29331]: mon.smithi029@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-18T01:10:55.267 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:10:55 smithi029 ceph-mon[29331]: mon.smithi029@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-18T01:10:55.267 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:10:55 smithi029 ceph-mon[29331]: mon.smithi029@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-18T01:10:55.267 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:10:55 smithi029 ceph-mon[29331]: mon.smithi029@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-03-18T01:10:55.267 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:10:55 smithi029 ceph-mon[29331]: mon.smithi029 is new leader, mons smithi029 in quorum (ranks 0) 2024-03-18T01:10:55.267 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:10:55 smithi029 ceph-mon[29331]: monmap e1: 1 mons at {smithi029=[v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-18T01:10:55.268 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:10:55 smithi029 ceph-mon[29331]: fsmap 2024-03-18T01:10:55.268 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:10:55 smithi029 ceph-mon[29331]: osdmap e1: 0 total, 0 up, 0 in 2024-03-18T01:10:55.268 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:10:55 smithi029 ceph-mon[29331]: mgrmap e1: no daemons active 2024-03-18T01:10:55.268 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:10:55 smithi029 bash[29312]: 0a7fff1d6dfcd675abb5a89188965e3ff6387fe9e1977d5eb7eafe0a990b5420 2024-03-18T01:10:55.268 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:10:55 smithi029 systemd[1]: Started Ceph mon.smithi029 for 37f51182-e4c4-11ee-95c9-87774f69a715. 2024-03-18T01:10:56.177 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:10:55.554+0000 7f2e9d99b700 1 Processor -- start 2024-03-18T01:10:56.177 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:10:55.554+0000 7f2e9d99b700 1 -- start start 2024-03-18T01:10:56.178 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:10:55.554+0000 7f2e9d99b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2e98076b20 0x7f2e981053b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:10:56.178 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:10:55.554+0000 7f2e9d99b700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e981058f0 con 0x7f2e98076b20 2024-03-18T01:10:56.178 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:10:55.555+0000 7f2e96ffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2e98076b20 0x7f2e981053b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:10:56.178 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:10:55.555+0000 7f2e96ffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2e98076b20 0x7f2e981053b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51846/0 (socket says 172.21.15.29:51846) 2024-03-18T01:10:56.178 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:10:55.555+0000 7f2e96ffd700 1 -- 172.21.15.29:0/1838374481 learned_addr learned my addr 172.21.15.29:0/1838374481 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-18T01:10:56.178 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:10:55.555+0000 7f2e96ffd700 1 -- 172.21.15.29:0/1838374481 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e98105a30 con 0x7f2e98076b20 2024-03-18T01:10:56.178 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:10:55.555+0000 7f2e96ffd700 1 --2- 172.21.15.29:0/1838374481 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2e98076b20 0x7f2e981053b0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f2e90009e50 tx=0x7f2e90009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=39e53cc80a5ea87f server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:10:56.178 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:10:55.555+0000 7f2e95ffb700 1 -- 172.21.15.29:0/1838374481 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2e900092e0 con 0x7f2e98076b20 2024-03-18T01:10:56.178 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:10:55.555+0000 7f2e95ffb700 1 -- 172.21.15.29:0/1838374481 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f2e90012430 con 0x7f2e98076b20 2024-03-18T01:10:56.178 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:10:55.555+0000 7f2e95ffb700 1 -- 172.21.15.29:0/1838374481 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2e900075c0 con 0x7f2e98076b20 2024-03-18T01:10:56.178 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:10:55.556+0000 7f2e9d99b700 1 -- 172.21.15.29:0/1838374481 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2e98076b20 msgr2=0x7f2e981053b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:10:56.178 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:10:55.556+0000 7f2e9d99b700 1 --2- 172.21.15.29:0/1838374481 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2e98076b20 0x7f2e981053b0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f2e90009e50 tx=0x7f2e90009840 comp rx=0 tx=0).stop 2024-03-18T01:10:56.178 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:10:55.556+0000 7f2e9d99b700 1 -- 172.21.15.29:0/1838374481 shutdown_connections 2024-03-18T01:10:56.178 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:10:55.556+0000 7f2e9d99b700 1 --2- 172.21.15.29:0/1838374481 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2e98076b20 0x7f2e981053b0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:10:56.179 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:10:55.556+0000 7f2e9d99b700 1 -- 172.21.15.29:0/1838374481 >> 172.21.15.29:0/1838374481 conn(0x7f2e980fcd80 msgr2=0x7f2e980ff190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:10:56.179 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:10:55.556+0000 7f2e9d99b700 1 -- 172.21.15.29:0/1838374481 shutdown_connections 2024-03-18T01:10:56.179 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:10:55.556+0000 7f2e9d99b700 1 -- 172.21.15.29:0/1838374481 wait complete. 2024-03-18T01:10:56.179 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:10:55.557+0000 7f2e9d99b700 1 Processor -- start 2024-03-18T01:10:56.179 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:10:55.557+0000 7f2e9d99b700 1 -- start start 2024-03-18T01:10:56.179 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:10:55.557+0000 7f2e9d99b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2e98076b20 0x7f2e981115d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:10:56.179 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:10:55.557+0000 7f2e9d99b700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e98111b10 con 0x7f2e98076b20 2024-03-18T01:10:56.179 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:10:55.557+0000 7f2e96ffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2e98076b20 0x7f2e981115d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:10:56.179 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:10:55.557+0000 7f2e96ffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2e98076b20 0x7f2e981115d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51850/0 (socket says 172.21.15.29:51850) 2024-03-18T01:10:56.179 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:10:55.557+0000 7f2e96ffd700 1 -- 172.21.15.29:0/3644391765 learned_addr learned my addr 172.21.15.29:0/3644391765 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-18T01:10:56.179 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:10:55.557+0000 7f2e96ffd700 1 -- 172.21.15.29:0/3644391765 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e90009ae0 con 0x7f2e98076b20 2024-03-18T01:10:56.179 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:10:55.557+0000 7f2e96ffd700 1 --2- 172.21.15.29:0/3644391765 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2e98076b20 0x7f2e981115d0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f2e90009fd0 tx=0x7f2e90009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:10:56.179 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:10:55.558+0000 7f2e8f7fe700 1 -- 172.21.15.29:0/3644391765 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2e900092e0 con 0x7f2e98076b20 2024-03-18T01:10:56.179 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:10:55.558+0000 7f2e8f7fe700 1 -- 172.21.15.29:0/3644391765 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f2e90007cc0 con 0x7f2e98076b20 2024-03-18T01:10:56.179 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:10:55.558+0000 7f2e9d99b700 1 -- 172.21.15.29:0/3644391765 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e98111d10 con 0x7f2e98076b20 2024-03-18T01:10:56.180 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:10:55.558+0000 7f2e9d99b700 1 -- 172.21.15.29:0/3644391765 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e98112880 con 0x7f2e98076b20 2024-03-18T01:10:56.180 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:10:55.558+0000 7f2e8f7fe700 1 -- 172.21.15.29:0/3644391765 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2e9001c880 con 0x7f2e98076b20 2024-03-18T01:10:56.180 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:10:55.558+0000 7f2e8f7fe700 1 -- 172.21.15.29:0/3644391765 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f2e90019070 con 0x7f2e98076b20 2024-03-18T01:10:56.180 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:10:55.558+0000 7f2e8f7fe700 1 -- 172.21.15.29:0/3644391765 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f2e90025ac0 con 0x7f2e98076b20 2024-03-18T01:10:56.180 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:10:55.559+0000 7f2e9d99b700 1 -- 172.21.15.29:0/3644391765 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e9805f5f0 con 0x7f2e98076b20 2024-03-18T01:10:56.180 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:10:55.561+0000 7f2e8f7fe700 1 -- 172.21.15.29:0/3644391765 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7f2e9002a060 con 0x7f2e98076b20 2024-03-18T01:10:56.180 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:10:55.629+0000 7f2e9d99b700 1 -- 172.21.15.29:0/3644391765 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) v1 -- 0x7f2e9804bed0 con 0x7f2e98076b20 2024-03-18T01:10:56.180 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:10:55.631+0000 7f2e8f7fe700 1 -- 172.21.15.29:0/3644391765 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) v1 ==== 124+0+0 (secure 0 0 0) 0x7f2e9804bed0 con 0x7f2e98076b20 2024-03-18T01:10:56.180 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:10:55.632+0000 7f2e9d99b700 1 -- 172.21.15.29:0/3644391765 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2e98076b20 msgr2=0x7f2e981115d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:10:56.180 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:10:55.632+0000 7f2e9d99b700 1 --2- 172.21.15.29:0/3644391765 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2e98076b20 0x7f2e981115d0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f2e90009fd0 tx=0x7f2e90009840 comp rx=0 tx=0).stop 2024-03-18T01:10:56.180 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:10:55.632+0000 7f2e9d99b700 1 -- 172.21.15.29:0/3644391765 shutdown_connections 2024-03-18T01:10:56.180 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:10:55.632+0000 7f2e9d99b700 1 --2- 172.21.15.29:0/3644391765 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2e98076b20 0x7f2e981115d0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:10:56.180 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:10:55.632+0000 7f2e9d99b700 1 -- 172.21.15.29:0/3644391765 >> 172.21.15.29:0/3644391765 conn(0x7f2e980fcd80 msgr2=0x7f2e980fd9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:10:56.180 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:10:55.632+0000 7f2e9d99b700 1 -- 172.21.15.29:0/3644391765 shutdown_connections 2024-03-18T01:10:56.180 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:10:55.632+0000 7f2e9d99b700 1 -- 172.21.15.29:0/3644391765 wait complete. 2024-03-18T01:10:56.181 INFO:teuthology.orchestra.run.smithi029.stdout:Wrote config to /etc/ceph/ceph.conf 2024-03-18T01:10:56.181 INFO:teuthology.orchestra.run.smithi029.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-03-18T01:10:56.181 INFO:teuthology.orchestra.run.smithi029.stdout:Creating mgr... 2024-03-18T01:10:56.181 INFO:teuthology.orchestra.run.smithi029.stdout:Verifying port 9283 ... 2024-03-18T01:10:56.355 INFO:teuthology.orchestra.run.smithi029.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-37f51182-e4c4-11ee-95c9-87774f69a715@mgr.smithi029.qviwhl 2024-03-18T01:10:56.355 INFO:teuthology.orchestra.run.smithi029.stdout:systemctl: stderr Failed to reset failed state of unit ceph-37f51182-e4c4-11ee-95c9-87774f69a715@mgr.smithi029.qviwhl.service: Unit ceph-37f51182-e4c4-11ee-95c9-87774f69a715@mgr.smithi029.qviwhl.service not loaded. 2024-03-18T01:10:56.509 INFO:teuthology.orchestra.run.smithi029.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-37f51182-e4c4-11ee-95c9-87774f69a715.target.wants/ceph-37f51182-e4c4-11ee-95c9-87774f69a715@mgr.smithi029.qviwhl.service → /etc/systemd/system/ceph-37f51182-e4c4-11ee-95c9-87774f69a715@.service. 2024-03-18T01:10:56.819 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:10:56 smithi029 ceph-mon[29331]: from='client.? 172.21.15.29:0/3644391765' entity='client.admin' 2024-03-18T01:10:57.042 INFO:teuthology.orchestra.run.smithi029.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-18T01:10:57.042 INFO:teuthology.orchestra.run.smithi029.stdout:systemctl: stdout disabled 2024-03-18T01:10:57.049 INFO:teuthology.orchestra.run.smithi029.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-18T01:10:57.049 INFO:teuthology.orchestra.run.smithi029.stdout:systemctl: stdout inactive 2024-03-18T01:10:57.049 INFO:teuthology.orchestra.run.smithi029.stdout:firewalld.service is not enabled 2024-03-18T01:10:57.049 INFO:teuthology.orchestra.run.smithi029.stdout:Not possible to enable service . firewalld.service is not available 2024-03-18T01:10:57.058 INFO:teuthology.orchestra.run.smithi029.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-18T01:10:57.058 INFO:teuthology.orchestra.run.smithi029.stdout:systemctl: stdout disabled 2024-03-18T01:10:57.065 INFO:teuthology.orchestra.run.smithi029.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-18T01:10:57.065 INFO:teuthology.orchestra.run.smithi029.stdout:systemctl: stdout inactive 2024-03-18T01:10:57.065 INFO:teuthology.orchestra.run.smithi029.stdout:firewalld.service is not enabled 2024-03-18T01:10:57.065 INFO:teuthology.orchestra.run.smithi029.stdout:Not possible to open ports <[9283]>. firewalld.service is not available 2024-03-18T01:10:57.065 INFO:teuthology.orchestra.run.smithi029.stdout:Waiting for mgr to start... 2024-03-18T01:10:57.066 INFO:teuthology.orchestra.run.smithi029.stdout:Waiting for mgr... 2024-03-18T01:10:57.819 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:10:57 smithi029 ceph-mon[29331]: from='client.? 172.21.15.29:0/2935327541' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-18T01:10:58.156 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout 2024-03-18T01:10:58.156 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout { 2024-03-18T01:10:58.156 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "fsid": "37f51182-e4c4-11ee-95c9-87774f69a715", 2024-03-18T01:10:58.157 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "health": { 2024-03-18T01:10:58.157 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-18T01:10:58.157 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-18T01:10:58.157 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-18T01:10:58.157 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout }, 2024-03-18T01:10:58.157 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-18T01:10:58.157 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-18T01:10:58.157 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout 0 2024-03-18T01:10:58.157 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout ], 2024-03-18T01:10:58.157 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-18T01:10:58.157 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "smithi029" 2024-03-18T01:10:58.157 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout ], 2024-03-18T01:10:58.157 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "quorum_age": 2, 2024-03-18T01:10:58.157 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-18T01:10:58.157 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-18T01:10:58.158 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "quincy", 2024-03-18T01:10:58.158 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-18T01:10:58.158 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout }, 2024-03-18T01:10:58.158 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-18T01:10:58.158 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-18T01:10:58.158 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-18T01:10:58.158 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-18T01:10:58.158 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-18T01:10:58.165 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-18T01:10:58.165 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-18T01:10:58.165 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-18T01:10:58.166 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout }, 2024-03-18T01:10:58.166 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-18T01:10:58.166 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-18T01:10:58.166 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-18T01:10:58.166 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-18T01:10:58.166 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-18T01:10:58.166 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-18T01:10:58.166 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-18T01:10:58.166 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-18T01:10:58.166 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-18T01:10:58.166 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout }, 2024-03-18T01:10:58.166 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-18T01:10:58.166 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-18T01:10:58.166 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-18T01:10:58.167 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-18T01:10:58.167 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout }, 2024-03-18T01:10:58.167 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-18T01:10:58.167 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-18T01:10:58.167 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-18T01:10:58.167 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-18T01:10:58.167 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-18T01:10:58.167 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-18T01:10:58.167 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "restful" 2024-03-18T01:10:58.167 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout ], 2024-03-18T01:10:58.167 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-18T01:10:58.167 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout }, 2024-03-18T01:10:58.167 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-18T01:10:58.167 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-18T01:10:58.167 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "modified": "2024-03-18T01:10:49.947500+0000", 2024-03-18T01:10:58.167 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-18T01:10:58.168 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout }, 2024-03-18T01:10:58.168 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-18T01:10:58.168 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout } 2024-03-18T01:10:58.168 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:10:57.532+0000 7f2fbf718700 1 Processor -- start 2024-03-18T01:10:58.168 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:10:57.532+0000 7f2fbf718700 1 -- start start 2024-03-18T01:10:58.168 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:10:57.532+0000 7f2fbf718700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2fb8102720 0x7f2fb8102b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:10:58.168 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:10:57.532+0000 7f2fbf718700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fb8103070 con 0x7f2fb8102720 2024-03-18T01:10:58.168 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:10:57.533+0000 7f2fbd4b4700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2fb8102720 0x7f2fb8102b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:10:58.168 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:10:57.533+0000 7f2fbd4b4700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2fb8102720 0x7f2fb8102b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51864/0 (socket says 172.21.15.29:51864) 2024-03-18T01:10:58.168 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:10:57.533+0000 7f2fbd4b4700 1 -- 172.21.15.29:0/197953460 learned_addr learned my addr 172.21.15.29:0/197953460 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-18T01:10:58.168 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:10:57.533+0000 7f2fbd4b4700 1 -- 172.21.15.29:0/197953460 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fb81031b0 con 0x7f2fb8102720 2024-03-18T01:10:58.168 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:10:57.533+0000 7f2fbd4b4700 1 --2- 172.21.15.29:0/197953460 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2fb8102720 0x7f2fb8102b30 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f2fa8009e50 tx=0x7f2fa8009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54797eac16f1fef7 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:10:58.168 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:10:57.533+0000 7f2fb7fff700 1 -- 172.21.15.29:0/197953460 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2fa80092e0 con 0x7f2fb8102720 2024-03-18T01:10:58.168 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:10:57.533+0000 7f2fb7fff700 1 -- 172.21.15.29:0/197953460 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f2fa8012430 con 0x7f2fb8102720 2024-03-18T01:10:58.168 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:10:57.533+0000 7f2fb7fff700 1 -- 172.21.15.29:0/197953460 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2fa80075c0 con 0x7f2fb8102720 2024-03-18T01:10:58.169 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:10:57.534+0000 7f2fbf718700 1 -- 172.21.15.29:0/197953460 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2fb8102720 msgr2=0x7f2fb8102b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:10:58.169 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:10:57.534+0000 7f2fbf718700 1 --2- 172.21.15.29:0/197953460 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2fb8102720 0x7f2fb8102b30 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f2fa8009e50 tx=0x7f2fa8009840 comp rx=0 tx=0).stop 2024-03-18T01:10:58.169 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:10:57.534+0000 7f2fbf718700 1 -- 172.21.15.29:0/197953460 shutdown_connections 2024-03-18T01:10:58.169 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:10:57.534+0000 7f2fbf718700 1 --2- 172.21.15.29:0/197953460 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2fb8102720 0x7f2fb8102b30 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:10:58.169 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:10:57.534+0000 7f2fbf718700 1 -- 172.21.15.29:0/197953460 >> 172.21.15.29:0/197953460 conn(0x7f2fb80fde40 msgr2=0x7f2fb81002b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:10:58.169 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:10:57.534+0000 7f2fbf718700 1 -- 172.21.15.29:0/197953460 shutdown_connections 2024-03-18T01:10:58.169 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:10:57.534+0000 7f2fbf718700 1 -- 172.21.15.29:0/197953460 wait complete. 2024-03-18T01:10:58.169 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:10:57.535+0000 7f2fbf718700 1 Processor -- start 2024-03-18T01:10:58.169 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:10:57.535+0000 7f2fbf718700 1 -- start start 2024-03-18T01:10:58.169 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:10:57.535+0000 7f2fbf718700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2fb8102720 0x7f2fb8111580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:10:58.169 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:10:57.535+0000 7f2fbf718700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fb8111ac0 con 0x7f2fb8102720 2024-03-18T01:10:58.169 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:10:57.535+0000 7f2fbd4b4700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2fb8102720 0x7f2fb8111580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:10:58.169 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:10:57.535+0000 7f2fbd4b4700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2fb8102720 0x7f2fb8111580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51870/0 (socket says 172.21.15.29:51870) 2024-03-18T01:10:58.169 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:10:57.535+0000 7f2fbd4b4700 1 -- 172.21.15.29:0/2935327541 learned_addr learned my addr 172.21.15.29:0/2935327541 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-18T01:10:58.169 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:10:57.535+0000 7f2fbd4b4700 1 -- 172.21.15.29:0/2935327541 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fa8009ae0 con 0x7f2fb8102720 2024-03-18T01:10:58.170 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:10:57.535+0000 7f2fbd4b4700 1 --2- 172.21.15.29:0/2935327541 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2fb8102720 0x7f2fb8111580 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f2fa8009fd0 tx=0x7f2fa8009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:10:58.170 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:10:57.535+0000 7f2fb5ffb700 1 -- 172.21.15.29:0/2935327541 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2fa80092e0 con 0x7f2fb8102720 2024-03-18T01:10:58.170 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:10:57.535+0000 7f2fb5ffb700 1 -- 172.21.15.29:0/2935327541 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f2fa8007cc0 con 0x7f2fb8102720 2024-03-18T01:10:58.170 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:10:57.535+0000 7f2fbf718700 1 -- 172.21.15.29:0/2935327541 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2fb8111cc0 con 0x7f2fb8102720 2024-03-18T01:10:58.170 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:10:57.535+0000 7f2fb5ffb700 1 -- 172.21.15.29:0/2935327541 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2fa801c9a0 con 0x7f2fb8102720 2024-03-18T01:10:58.170 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:10:57.536+0000 7f2fbf718700 1 -- 172.21.15.29:0/2935327541 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2fb8112830 con 0x7f2fb8102720 2024-03-18T01:10:58.170 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:10:57.536+0000 7f2fb5ffb700 1 -- 172.21.15.29:0/2935327541 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f2fa8019070 con 0x7f2fb8102720 2024-03-18T01:10:58.170 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:10:57.536+0000 7f2fb5ffb700 1 -- 172.21.15.29:0/2935327541 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f2fa8025b10 con 0x7f2fb8102720 2024-03-18T01:10:58.170 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:10:57.537+0000 7f2fbf718700 1 -- 172.21.15.29:0/2935327541 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2fb805f5f0 con 0x7f2fb8102720 2024-03-18T01:10:58.170 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:10:57.539+0000 7f2fb5ffb700 1 -- 172.21.15.29:0/2935327541 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7f2fa802a060 con 0x7f2fb8102720 2024-03-18T01:10:58.170 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:10:57.612+0000 7f2fbf718700 1 -- 172.21.15.29:0/2935327541 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f2fb804bed0 con 0x7f2fb8102720 2024-03-18T01:10:58.170 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:10:57.613+0000 7f2fb5ffb700 1 -- 172.21.15.29:0/2935327541 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1248 (secure 0 0 0) 0x7f2fa8020070 con 0x7f2fb8102720 2024-03-18T01:10:58.170 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:10:57.614+0000 7f2fbf718700 1 -- 172.21.15.29:0/2935327541 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2fb8102720 msgr2=0x7f2fb8111580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:10:58.170 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:10:57.614+0000 7f2fbf718700 1 --2- 172.21.15.29:0/2935327541 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2fb8102720 0x7f2fb8111580 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f2fa8009fd0 tx=0x7f2fa8009840 comp rx=0 tx=0).stop 2024-03-18T01:10:58.171 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:10:57.614+0000 7f2fbf718700 1 -- 172.21.15.29:0/2935327541 shutdown_connections 2024-03-18T01:10:58.171 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:10:57.614+0000 7f2fbf718700 1 --2- 172.21.15.29:0/2935327541 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2fb8102720 0x7f2fb8111580 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:10:58.171 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:10:57.614+0000 7f2fbf718700 1 -- 172.21.15.29:0/2935327541 >> 172.21.15.29:0/2935327541 conn(0x7f2fb80fde40 msgr2=0x7f2fb80fea10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:10:58.171 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:10:57.614+0000 7f2fbf718700 1 -- 172.21.15.29:0/2935327541 shutdown_connections 2024-03-18T01:10:58.171 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:10:57.614+0000 7f2fbf718700 1 -- 172.21.15.29:0/2935327541 wait complete. 2024-03-18T01:10:58.171 INFO:teuthology.orchestra.run.smithi029.stdout:mgr not available, waiting (1/15)... 2024-03-18T01:11:04.030 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:11:03 smithi029 ceph-mon[29331]: Activating manager daemon smithi029.qviwhl 2024-03-18T01:11:04.030 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:11:03 smithi029 ceph-mon[29331]: mgrmap e2: smithi029.qviwhl(active, starting, since 0.00264821s) 2024-03-18T01:11:04.031 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:11:03 smithi029 ceph-mon[29331]: from='mgr.14100 172.21.15.29:0/3042144736' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-18T01:11:04.031 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:11:03 smithi029 ceph-mon[29331]: from='mgr.14100 172.21.15.29:0/3042144736' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-18T01:11:04.031 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:11:03 smithi029 ceph-mon[29331]: from='mgr.14100 172.21.15.29:0/3042144736' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-18T01:11:04.031 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:11:03 smithi029 ceph-mon[29331]: from='mgr.14100 172.21.15.29:0/3042144736' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "mon metadata", "id": "smithi029"}]: dispatch 2024-03-18T01:11:04.031 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:11:03 smithi029 ceph-mon[29331]: from='mgr.14100 172.21.15.29:0/3042144736' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "mgr metadata", "who": "smithi029.qviwhl", "id": "smithi029.qviwhl"}]: dispatch 2024-03-18T01:11:04.031 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:11:03 smithi029 ceph-mon[29331]: Manager daemon smithi029.qviwhl is now available 2024-03-18T01:11:04.031 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:11:03 smithi029 ceph-mon[29331]: from='mgr.14100 172.21.15.29:0/3042144736' entity='mgr.smithi029.qviwhl' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi029.qviwhl/mirror_snapshot_schedule"}]: dispatch 2024-03-18T01:11:04.031 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:11:03 smithi029 ceph-mon[29331]: from='mgr.14100 172.21.15.29:0/3042144736' entity='mgr.smithi029.qviwhl' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi029.qviwhl/trash_purge_schedule"}]: dispatch 2024-03-18T01:11:04.031 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:11:03 smithi029 ceph-mon[29331]: from='mgr.14100 172.21.15.29:0/3042144736' entity='mgr.smithi029.qviwhl' 2024-03-18T01:11:04.031 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:11:03 smithi029 ceph-mon[29331]: from='mgr.14100 172.21.15.29:0/3042144736' entity='mgr.smithi029.qviwhl' 2024-03-18T01:11:04.031 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:11:03 smithi029 ceph-mon[29331]: from='mgr.14100 172.21.15.29:0/3042144736' entity='mgr.smithi029.qviwhl' 2024-03-18T01:11:05.819 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:11:05 smithi029 ceph-mon[29331]: mgrmap e3: smithi029.qviwhl(active, since 1.00728s) 2024-03-18T01:11:05.819 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:11:05 smithi029 ceph-mon[29331]: from='client.? 172.21.15.29:0/67027268' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-18T01:11:08.069 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:11:07 smithi029 ceph-mon[29331]: mgrmap e4: smithi029.qviwhl(active, since 3s) 2024-03-18T01:11:15.570 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout 2024-03-18T01:11:15.570 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout { 2024-03-18T01:11:15.570 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "fsid": "37f51182-e4c4-11ee-95c9-87774f69a715", 2024-03-18T01:11:15.570 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "health": { 2024-03-18T01:11:15.570 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-18T01:11:15.571 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-18T01:11:15.571 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-18T01:11:15.571 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout }, 2024-03-18T01:11:15.571 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-18T01:11:15.571 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-18T01:11:15.571 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout 0 2024-03-18T01:11:15.572 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout ], 2024-03-18T01:11:15.572 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-18T01:11:15.572 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "smithi029" 2024-03-18T01:11:15.572 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout ], 2024-03-18T01:11:15.572 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "quorum_age": 10, 2024-03-18T01:11:15.572 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-18T01:11:15.573 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-18T01:11:15.573 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "quincy", 2024-03-18T01:11:15.573 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-18T01:11:15.573 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout }, 2024-03-18T01:11:15.573 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-18T01:11:15.573 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-18T01:11:15.573 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-18T01:11:15.574 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-18T01:11:15.574 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-18T01:11:15.574 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-18T01:11:15.574 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-18T01:11:15.574 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-18T01:11:15.574 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout }, 2024-03-18T01:11:15.575 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-18T01:11:15.575 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-18T01:11:15.575 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-18T01:11:15.575 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-18T01:11:15.575 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-18T01:11:15.575 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-18T01:11:15.576 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-18T01:11:15.576 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-18T01:11:15.576 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-18T01:11:15.576 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout }, 2024-03-18T01:11:15.576 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-18T01:11:15.576 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-18T01:11:15.577 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-18T01:11:15.577 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-18T01:11:15.577 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout }, 2024-03-18T01:11:15.577 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-18T01:11:15.577 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-18T01:11:15.577 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-18T01:11:15.578 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-18T01:11:15.578 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-18T01:11:15.578 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-18T01:11:15.578 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "restful" 2024-03-18T01:11:15.578 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout ], 2024-03-18T01:11:15.578 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-18T01:11:15.578 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout }, 2024-03-18T01:11:15.579 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-18T01:11:15.579 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-18T01:11:15.579 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "modified": "2024-03-18T01:10:49.947500+0000", 2024-03-18T01:11:15.579 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-18T01:11:15.579 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout }, 2024-03-18T01:11:15.579 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-18T01:11:15.580 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout } 2024-03-18T01:11:15.580 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:05.234+0000 7f2c1fe1f700 1 Processor -- start 2024-03-18T01:11:15.580 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:05.234+0000 7f2c1fe1f700 1 -- start start 2024-03-18T01:11:15.580 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:05.234+0000 7f2c1fe1f700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2c18102720 0x7f2c18102b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:11:15.580 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:05.235+0000 7f2c1fe1f700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c18103070 con 0x7f2c18102720 2024-03-18T01:11:15.580 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:05.235+0000 7f2c1dbbb700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2c18102720 0x7f2c18102b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:11:15.581 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:05.235+0000 7f2c1dbbb700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2c18102720 0x7f2c18102b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52246/0 (socket says 172.21.15.29:52246) 2024-03-18T01:11:15.581 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:05.235+0000 7f2c1dbbb700 1 -- 172.21.15.29:0/3972321183 learned_addr learned my addr 172.21.15.29:0/3972321183 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-18T01:11:15.581 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:05.235+0000 7f2c1dbbb700 1 -- 172.21.15.29:0/3972321183 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c181031b0 con 0x7f2c18102720 2024-03-18T01:11:15.581 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:05.235+0000 7f2c1dbbb700 1 --2- 172.21.15.29:0/3972321183 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2c18102720 0x7f2c18102b30 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f2c080098e0 tx=0x7f2c08009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e465ed79adff8e09 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:11:15.581 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:05.235+0000 7f2c1cbb9700 1 -- 172.21.15.29:0/3972321183 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2c0800b860 con 0x7f2c18102720 2024-03-18T01:11:15.581 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:05.235+0000 7f2c1cbb9700 1 -- 172.21.15.29:0/3972321183 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f2c0800be50 con 0x7f2c18102720 2024-03-18T01:11:15.582 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:05.235+0000 7f2c1cbb9700 1 -- 172.21.15.29:0/3972321183 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2c08010380 con 0x7f2c18102720 2024-03-18T01:11:15.582 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:05.236+0000 7f2c1fe1f700 1 -- 172.21.15.29:0/3972321183 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2c18102720 msgr2=0x7f2c18102b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:11:15.582 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:05.236+0000 7f2c1fe1f700 1 --2- 172.21.15.29:0/3972321183 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2c18102720 0x7f2c18102b30 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f2c080098e0 tx=0x7f2c08009250 comp rx=0 tx=0).stop 2024-03-18T01:11:15.582 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:05.236+0000 7f2c1fe1f700 1 -- 172.21.15.29:0/3972321183 shutdown_connections 2024-03-18T01:11:15.582 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:05.236+0000 7f2c1fe1f700 1 --2- 172.21.15.29:0/3972321183 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2c18102720 0x7f2c18102b30 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:11:15.582 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:05.236+0000 7f2c1fe1f700 1 -- 172.21.15.29:0/3972321183 >> 172.21.15.29:0/3972321183 conn(0x7f2c180fdee0 msgr2=0x7f2c181002f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:11:15.583 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:05.236+0000 7f2c1fe1f700 1 -- 172.21.15.29:0/3972321183 shutdown_connections 2024-03-18T01:11:15.583 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:05.236+0000 7f2c1fe1f700 1 -- 172.21.15.29:0/3972321183 wait complete. 2024-03-18T01:11:15.583 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:05.236+0000 7f2c1fe1f700 1 Processor -- start 2024-03-18T01:11:15.583 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:05.236+0000 7f2c1fe1f700 1 -- start start 2024-03-18T01:11:15.583 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:05.237+0000 7f2c1fe1f700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2c18102720 0x7f2c18078bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:11:15.583 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:05.237+0000 7f2c1fe1f700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c18079110 con 0x7f2c18102720 2024-03-18T01:11:15.583 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:05.237+0000 7f2c1dbbb700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2c18102720 0x7f2c18078bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:11:15.584 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:05.237+0000 7f2c1dbbb700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2c18102720 0x7f2c18078bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52256/0 (socket says 172.21.15.29:52256) 2024-03-18T01:11:15.584 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:05.237+0000 7f2c1dbbb700 1 -- 172.21.15.29:0/67027268 learned_addr learned my addr 172.21.15.29:0/67027268 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-18T01:11:15.584 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:05.237+0000 7f2c1dbbb700 1 -- 172.21.15.29:0/67027268 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c080094f0 con 0x7f2c18102720 2024-03-18T01:11:15.584 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:05.237+0000 7f2c1dbbb700 1 --2- 172.21.15.29:0/67027268 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2c18102720 0x7f2c18078bd0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f2c0800b440 tx=0x7f2c08009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:11:15.584 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:05.237+0000 7f2c167fc700 1 -- 172.21.15.29:0/67027268 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2c08010760 con 0x7f2c18102720 2024-03-18T01:11:15.584 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:05.237+0000 7f2c167fc700 1 -- 172.21.15.29:0/67027268 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f2c08010d50 con 0x7f2c18102720 2024-03-18T01:11:15.585 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:05.237+0000 7f2c1fe1f700 1 -- 172.21.15.29:0/67027268 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c18077180 con 0x7f2c18102720 2024-03-18T01:11:15.585 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:05.237+0000 7f2c167fc700 1 -- 172.21.15.29:0/67027268 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2c0801ae20 con 0x7f2c18102720 2024-03-18T01:11:15.585 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:05.237+0000 7f2c1fe1f700 1 -- 172.21.15.29:0/67027268 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c18077670 con 0x7f2c18102720 2024-03-18T01:11:15.585 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:05.238+0000 7f2c167fc700 1 -- 172.21.15.29:0/67027268 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 3) v1 ==== 43251+0+0 (secure 0 0 0) 0x7f2c080108c0 con 0x7f2c18102720 2024-03-18T01:11:15.585 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:05.238+0000 7f2c1fe1f700 1 -- 172.21.15.29:0/67027268 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c1804bed0 con 0x7f2c18102720 2024-03-18T01:11:15.585 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:05.238+0000 7f2c167fc700 1 --2- 172.21.15.29:0/67027268 >> [v2:172.21.15.29:6800/1702596206,v1:172.21.15.29:6801/1702596206] conn(0x7f2c00036660 0x7f2c00038b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:11:15.586 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:05.238+0000 7f2c167fc700 1 -- 172.21.15.29:0/67027268 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f2c0804a240 con 0x7f2c18102720 2024-03-18T01:11:15.586 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:05.238+0000 7f2c1d3ba700 1 --2- 172.21.15.29:0/67027268 >> [v2:172.21.15.29:6800/1702596206,v1:172.21.15.29:6801/1702596206] conn(0x7f2c00036660 0x7f2c00038b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:11:15.586 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:05.239+0000 7f2c1d3ba700 1 --2- 172.21.15.29:0/67027268 >> [v2:172.21.15.29:6800/1702596206,v1:172.21.15.29:6801/1702596206] conn(0x7f2c00036660 0x7f2c00038b10 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f2c0c009990 tx=0x7f2c0c006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:11:15.586 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:05.243+0000 7f2c167fc700 1 -- 172.21.15.29:0/67027268 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175520 (secure 0 0 0) 0x7f2c080187c0 con 0x7f2c18102720 2024-03-18T01:11:15.586 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:05.512+0000 7f2c1fe1f700 1 -- 172.21.15.29:0/67027268 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f2c18104540 con 0x7f2c18102720 2024-03-18T01:11:15.587 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:05.512+0000 7f2c167fc700 1 -- 172.21.15.29:0/67027268 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1248 (secure 0 0 0) 0x7f2c08019e60 con 0x7f2c18102720 2024-03-18T01:11:15.587 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:05.514+0000 7f2c1fe1f700 1 -- 172.21.15.29:0/67027268 >> [v2:172.21.15.29:6800/1702596206,v1:172.21.15.29:6801/1702596206] conn(0x7f2c00036660 msgr2=0x7f2c00038b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:11:15.587 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:05.514+0000 7f2c1fe1f700 1 --2- 172.21.15.29:0/67027268 >> [v2:172.21.15.29:6800/1702596206,v1:172.21.15.29:6801/1702596206] conn(0x7f2c00036660 0x7f2c00038b10 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f2c0c009990 tx=0x7f2c0c006e40 comp rx=0 tx=0).stop 2024-03-18T01:11:15.587 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:05.514+0000 7f2c1fe1f700 1 -- 172.21.15.29:0/67027268 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2c18102720 msgr2=0x7f2c18078bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:11:15.587 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:05.514+0000 7f2c1fe1f700 1 --2- 172.21.15.29:0/67027268 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2c18102720 0x7f2c18078bd0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f2c0800b440 tx=0x7f2c08009250 comp rx=0 tx=0).stop 2024-03-18T01:11:15.588 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:05.514+0000 7f2c1fe1f700 1 -- 172.21.15.29:0/67027268 shutdown_connections 2024-03-18T01:11:15.588 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:05.514+0000 7f2c1fe1f700 1 --2- 172.21.15.29:0/67027268 >> [v2:172.21.15.29:6800/1702596206,v1:172.21.15.29:6801/1702596206] conn(0x7f2c00036660 0x7f2c00038b10 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:11:15.588 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:05.514+0000 7f2c1fe1f700 1 --2- 172.21.15.29:0/67027268 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2c18102720 0x7f2c18078bd0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:11:15.588 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:05.514+0000 7f2c1fe1f700 1 -- 172.21.15.29:0/67027268 >> 172.21.15.29:0/67027268 conn(0x7f2c180fdee0 msgr2=0x7f2c180fea60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:11:15.588 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:05.514+0000 7f2c1fe1f700 1 -- 172.21.15.29:0/67027268 shutdown_connections 2024-03-18T01:11:15.589 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:05.514+0000 7f2c1fe1f700 1 -- 172.21.15.29:0/67027268 wait complete. 2024-03-18T01:11:15.589 INFO:teuthology.orchestra.run.smithi029.stdout:mgr is available 2024-03-18T01:11:17.658 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:11:17 smithi029 ceph-mon[29331]: from='client.? 172.21.15.29:0/3312956177' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-03-18T01:11:19.007 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout 2024-03-18T01:11:19.007 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout [global] 2024-03-18T01:11:19.007 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout fsid = 37f51182-e4c4-11ee-95c9-87774f69a715 2024-03-18T01:11:19.007 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-03-18T01:11:19.007 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-03-18T01:11:19.007 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-03-18T01:11:19.007 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-03-18T01:11:19.007 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout 2024-03-18T01:11:19.007 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout [mgr] 2024-03-18T01:11:19.007 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-03-18T01:11:19.007 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout 2024-03-18T01:11:19.007 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout [osd] 2024-03-18T01:11:19.008 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-03-18T01:11:19.008 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-03-18T01:11:19.008 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-03-18T01:11:19.008 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-03-18T01:11:19.008 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-03-18T01:11:19.008 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:17.101+0000 7fd8fc413700 1 Processor -- start 2024-03-18T01:11:19.008 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:17.101+0000 7fd8fc413700 1 -- start start 2024-03-18T01:11:19.008 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:17.102+0000 7fd8fc413700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd8f41016e0 0x7fd8f4101af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:11:19.008 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:17.102+0000 7fd8fc413700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8f4102030 con 0x7fd8f41016e0 2024-03-18T01:11:19.008 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:17.102+0000 7fd8fa1af700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd8f41016e0 0x7fd8f4101af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:11:19.008 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:17.102+0000 7fd8fa1af700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd8f41016e0 0x7fd8f4101af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56334/0 (socket says 172.21.15.29:56334) 2024-03-18T01:11:19.008 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:17.102+0000 7fd8fa1af700 1 -- 172.21.15.29:0/4239188041 learned_addr learned my addr 172.21.15.29:0/4239188041 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-18T01:11:19.008 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:17.102+0000 7fd8fa1af700 1 -- 172.21.15.29:0/4239188041 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8f4102170 con 0x7fd8f41016e0 2024-03-18T01:11:19.008 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:17.102+0000 7fd8fa1af700 1 --2- 172.21.15.29:0/4239188041 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd8f41016e0 0x7fd8f4101af0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fd8f00098e0 tx=0x7fd8f0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a39311f50ab644cc server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:11:19.008 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:17.102+0000 7fd8f91ad700 1 -- 172.21.15.29:0/4239188041 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd8f000b860 con 0x7fd8f41016e0 2024-03-18T01:11:19.009 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:17.103+0000 7fd8f91ad700 1 -- 172.21.15.29:0/4239188041 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7fd8f000be50 con 0x7fd8f41016e0 2024-03-18T01:11:19.009 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:17.103+0000 7fd8f91ad700 1 -- 172.21.15.29:0/4239188041 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd8f0010380 con 0x7fd8f41016e0 2024-03-18T01:11:19.009 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:17.103+0000 7fd8fc413700 1 -- 172.21.15.29:0/4239188041 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd8f41016e0 msgr2=0x7fd8f4101af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:11:19.009 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:17.103+0000 7fd8fc413700 1 --2- 172.21.15.29:0/4239188041 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd8f41016e0 0x7fd8f4101af0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fd8f00098e0 tx=0x7fd8f0009250 comp rx=0 tx=0).stop 2024-03-18T01:11:19.009 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:17.103+0000 7fd8fc413700 1 -- 172.21.15.29:0/4239188041 shutdown_connections 2024-03-18T01:11:19.009 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:17.103+0000 7fd8fc413700 1 --2- 172.21.15.29:0/4239188041 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd8f41016e0 0x7fd8f4101af0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:11:19.009 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:17.103+0000 7fd8fc413700 1 -- 172.21.15.29:0/4239188041 >> 172.21.15.29:0/4239188041 conn(0x7fd8f40fce40 msgr2=0x7fd8f40ff270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:11:19.009 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:17.103+0000 7fd8fc413700 1 -- 172.21.15.29:0/4239188041 shutdown_connections 2024-03-18T01:11:19.009 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:17.103+0000 7fd8fc413700 1 -- 172.21.15.29:0/4239188041 wait complete. 2024-03-18T01:11:19.009 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:17.104+0000 7fd8fc413700 1 Processor -- start 2024-03-18T01:11:19.009 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:17.104+0000 7fd8fc413700 1 -- start start 2024-03-18T01:11:19.009 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:17.104+0000 7fd8fc413700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd8f41016e0 0x7fd8f4107870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:11:19.009 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:17.104+0000 7fd8fc413700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8f4107db0 con 0x7fd8f41016e0 2024-03-18T01:11:19.009 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:17.104+0000 7fd8fa1af700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd8f41016e0 0x7fd8f4107870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:11:19.010 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:17.104+0000 7fd8fa1af700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd8f41016e0 0x7fd8f4107870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56340/0 (socket says 172.21.15.29:56340) 2024-03-18T01:11:19.010 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:17.104+0000 7fd8fa1af700 1 -- 172.21.15.29:0/3312956177 learned_addr learned my addr 172.21.15.29:0/3312956177 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-18T01:11:19.010 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:17.104+0000 7fd8fa1af700 1 -- 172.21.15.29:0/3312956177 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8f00094f0 con 0x7fd8f41016e0 2024-03-18T01:11:19.010 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:17.104+0000 7fd8fa1af700 1 --2- 172.21.15.29:0/3312956177 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd8f41016e0 0x7fd8f4107870 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fd8f0006b00 tx=0x7fd8f0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:11:19.010 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:17.105+0000 7fd8eaffd700 1 -- 172.21.15.29:0/3312956177 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd8f0010760 con 0x7fd8f41016e0 2024-03-18T01:11:19.010 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:17.105+0000 7fd8eaffd700 1 -- 172.21.15.29:0/3312956177 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7fd8f0010d50 con 0x7fd8f41016e0 2024-03-18T01:11:19.010 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:17.105+0000 7fd8fc413700 1 -- 172.21.15.29:0/3312956177 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8f4105e70 con 0x7fd8f41016e0 2024-03-18T01:11:19.010 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:17.105+0000 7fd8eaffd700 1 -- 172.21.15.29:0/3312956177 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd8f001ad40 con 0x7fd8f41016e0 2024-03-18T01:11:19.010 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:17.105+0000 7fd8fc413700 1 -- 172.21.15.29:0/3312956177 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8f4106360 con 0x7fd8f41016e0 2024-03-18T01:11:19.010 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:17.105+0000 7fd8eaffd700 1 -- 172.21.15.29:0/3312956177 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 4) v1 ==== 43357+0+0 (secure 0 0 0) 0x7fd8f00108c0 con 0x7fd8f41016e0 2024-03-18T01:11:19.010 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:17.106+0000 7fd8fc413700 1 -- 172.21.15.29:0/3312956177 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8f404bed0 con 0x7fd8f41016e0 2024-03-18T01:11:19.010 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:17.106+0000 7fd8eaffd700 1 --2- 172.21.15.29:0/3312956177 >> [v2:172.21.15.29:6800/1702596206,v1:172.21.15.29:6801/1702596206] conn(0x7fd8e0036790 0x7fd8e0038c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:11:19.010 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:17.106+0000 7fd8eaffd700 1 -- 172.21.15.29:0/3312956177 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7fd8f004a4f0 con 0x7fd8f41016e0 2024-03-18T01:11:19.010 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:17.106+0000 7fd8f99ae700 1 --2- 172.21.15.29:0/3312956177 >> [v2:172.21.15.29:6800/1702596206,v1:172.21.15.29:6801/1702596206] conn(0x7fd8e0036790 0x7fd8e0038c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:11:19.011 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:17.106+0000 7fd8f99ae700 1 --2- 172.21.15.29:0/3312956177 >> [v2:172.21.15.29:6800/1702596206,v1:172.21.15.29:6801/1702596206] conn(0x7fd8e0036790 0x7fd8e0038c40 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fd8e4009990 tx=0x7fd8e4006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:11:19.011 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:17.111+0000 7fd8eaffd700 1 -- 172.21.15.29:0/3312956177 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175520 (secure 0 0 0) 0x7fd8f0018920 con 0x7fd8f41016e0 2024-03-18T01:11:19.011 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:17.300+0000 7fd8fc413700 1 -- 172.21.15.29:0/3312956177 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7fd8f410b330 con 0x7fd8f41016e0 2024-03-18T01:11:19.011 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:17.301+0000 7fd8eaffd700 1 -- 172.21.15.29:0/3312956177 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) v1 ==== 70+0+373 (secure 0 0 0) 0x7fd8f0018350 con 0x7fd8f41016e0 2024-03-18T01:11:19.011 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:17.303+0000 7fd8fc413700 1 -- 172.21.15.29:0/3312956177 >> [v2:172.21.15.29:6800/1702596206,v1:172.21.15.29:6801/1702596206] conn(0x7fd8e0036790 msgr2=0x7fd8e0038c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:11:19.011 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:17.303+0000 7fd8fc413700 1 --2- 172.21.15.29:0/3312956177 >> [v2:172.21.15.29:6800/1702596206,v1:172.21.15.29:6801/1702596206] conn(0x7fd8e0036790 0x7fd8e0038c40 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fd8e4009990 tx=0x7fd8e4006e40 comp rx=0 tx=0).stop 2024-03-18T01:11:19.011 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:17.303+0000 7fd8fc413700 1 -- 172.21.15.29:0/3312956177 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd8f41016e0 msgr2=0x7fd8f4107870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:11:19.011 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:17.303+0000 7fd8fc413700 1 --2- 172.21.15.29:0/3312956177 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd8f41016e0 0x7fd8f4107870 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fd8f0006b00 tx=0x7fd8f0009250 comp rx=0 tx=0).stop 2024-03-18T01:11:19.011 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:17.303+0000 7fd8fc413700 1 -- 172.21.15.29:0/3312956177 shutdown_connections 2024-03-18T01:11:19.011 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:17.303+0000 7fd8fc413700 1 --2- 172.21.15.29:0/3312956177 >> [v2:172.21.15.29:6800/1702596206,v1:172.21.15.29:6801/1702596206] conn(0x7fd8e0036790 0x7fd8e0038c40 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:11:19.011 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:17.303+0000 7fd8fc413700 1 --2- 172.21.15.29:0/3312956177 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd8f41016e0 0x7fd8f4107870 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:11:19.011 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:17.303+0000 7fd8fc413700 1 -- 172.21.15.29:0/3312956177 >> 172.21.15.29:0/3312956177 conn(0x7fd8f40fce40 msgr2=0x7fd8f40fdd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:11:19.011 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:17.303+0000 7fd8fc413700 1 -- 172.21.15.29:0/3312956177 shutdown_connections 2024-03-18T01:11:19.011 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:17.303+0000 7fd8fc413700 1 -- 172.21.15.29:0/3312956177 wait complete. 2024-03-18T01:11:19.011 INFO:teuthology.orchestra.run.smithi029.stdout:Enabling cephadm module... 2024-03-18T01:11:20.041 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:11:19 smithi029 ceph-mon[29331]: from='client.? 172.21.15.29:0/1686181815' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-03-18T01:11:20.326 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:19.429+0000 7f6e01261700 1 Processor -- start 2024-03-18T01:11:20.326 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:19.429+0000 7f6e01261700 1 -- start start 2024-03-18T01:11:20.327 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:19.429+0000 7f6e01261700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6dfc1015b0 0x7f6dfc1019c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:11:20.327 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:19.429+0000 7f6e01261700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6dfc101f00 con 0x7f6dfc1015b0 2024-03-18T01:11:20.327 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:19.429+0000 7f6dfaffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6dfc1015b0 0x7f6dfc1019c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:11:20.327 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:19.429+0000 7f6dfaffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6dfc1015b0 0x7f6dfc1019c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56348/0 (socket says 172.21.15.29:56348) 2024-03-18T01:11:20.327 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:19.429+0000 7f6dfaffd700 1 -- 172.21.15.29:0/2563810004 learned_addr learned my addr 172.21.15.29:0/2563810004 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-18T01:11:20.327 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:19.429+0000 7f6dfaffd700 1 -- 172.21.15.29:0/2563810004 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6dfc102040 con 0x7f6dfc1015b0 2024-03-18T01:11:20.327 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:19.430+0000 7f6dfaffd700 1 --2- 172.21.15.29:0/2563810004 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6dfc1015b0 0x7f6dfc1019c0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f6dec0098e0 tx=0x7f6dec009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14957c30f1c34fee server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:11:20.327 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:19.430+0000 7f6df9ffb700 1 -- 172.21.15.29:0/2563810004 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6dec00b860 con 0x7f6dfc1015b0 2024-03-18T01:11:20.327 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:19.430+0000 7f6df9ffb700 1 -- 172.21.15.29:0/2563810004 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f6dec00be50 con 0x7f6dfc1015b0 2024-03-18T01:11:20.327 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:19.430+0000 7f6df9ffb700 1 -- 172.21.15.29:0/2563810004 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6dec010380 con 0x7f6dfc1015b0 2024-03-18T01:11:20.327 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:19.430+0000 7f6e01261700 1 -- 172.21.15.29:0/2563810004 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6dfc1015b0 msgr2=0x7f6dfc1019c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:11:20.327 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:19.430+0000 7f6e01261700 1 --2- 172.21.15.29:0/2563810004 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6dfc1015b0 0x7f6dfc1019c0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f6dec0098e0 tx=0x7f6dec009250 comp rx=0 tx=0).stop 2024-03-18T01:11:20.327 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:19.430+0000 7f6e01261700 1 -- 172.21.15.29:0/2563810004 shutdown_connections 2024-03-18T01:11:20.327 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:19.430+0000 7f6e01261700 1 --2- 172.21.15.29:0/2563810004 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6dfc1015b0 0x7f6dfc1019c0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:11:20.328 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:19.430+0000 7f6e01261700 1 -- 172.21.15.29:0/2563810004 >> 172.21.15.29:0/2563810004 conn(0x7f6dfc0fcd70 msgr2=0x7f6dfc0ff180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:11:20.328 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:19.430+0000 7f6e01261700 1 -- 172.21.15.29:0/2563810004 shutdown_connections 2024-03-18T01:11:20.328 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:19.430+0000 7f6e01261700 1 -- 172.21.15.29:0/2563810004 wait complete. 2024-03-18T01:11:20.328 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:19.431+0000 7f6e01261700 1 Processor -- start 2024-03-18T01:11:20.328 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:19.431+0000 7f6e01261700 1 -- start start 2024-03-18T01:11:20.328 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:19.431+0000 7f6e01261700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6dfc1015b0 0x7f6dfc110490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:11:20.328 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:19.431+0000 7f6e01261700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6dfc1109d0 con 0x7f6dfc1015b0 2024-03-18T01:11:20.328 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:19.431+0000 7f6dfaffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6dfc1015b0 0x7f6dfc110490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:11:20.328 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:19.431+0000 7f6dfaffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6dfc1015b0 0x7f6dfc110490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56356/0 (socket says 172.21.15.29:56356) 2024-03-18T01:11:20.328 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:19.431+0000 7f6dfaffd700 1 -- 172.21.15.29:0/1686181815 learned_addr learned my addr 172.21.15.29:0/1686181815 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-18T01:11:20.328 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:19.432+0000 7f6dfaffd700 1 -- 172.21.15.29:0/1686181815 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6dec0094f0 con 0x7f6dfc1015b0 2024-03-18T01:11:20.328 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:19.432+0000 7f6dfaffd700 1 --2- 172.21.15.29:0/1686181815 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6dfc1015b0 0x7f6dfc110490 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f6dec000c00 tx=0x7f6dec009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:11:20.328 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:19.432+0000 7f6de37fe700 1 -- 172.21.15.29:0/1686181815 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6dec0107c0 con 0x7f6dfc1015b0 2024-03-18T01:11:20.328 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:19.432+0000 7f6de37fe700 1 -- 172.21.15.29:0/1686181815 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f6dec010db0 con 0x7f6dfc1015b0 2024-03-18T01:11:20.328 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:19.432+0000 7f6e01261700 1 -- 172.21.15.29:0/1686181815 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6dfc110bd0 con 0x7f6dfc1015b0 2024-03-18T01:11:20.329 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:19.432+0000 7f6de37fe700 1 -- 172.21.15.29:0/1686181815 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6dec01ad00 con 0x7f6dfc1015b0 2024-03-18T01:11:20.329 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:19.432+0000 7f6e01261700 1 -- 172.21.15.29:0/1686181815 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6dfc111740 con 0x7f6dfc1015b0 2024-03-18T01:11:20.329 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:19.433+0000 7f6de37fe700 1 -- 172.21.15.29:0/1686181815 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 4) v1 ==== 43357+0+0 (secure 0 0 0) 0x7f6dec0193c0 con 0x7f6dfc1015b0 2024-03-18T01:11:20.329 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:19.433+0000 7f6e01261700 1 -- 172.21.15.29:0/1686181815 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6dfc05f5f0 con 0x7f6dfc1015b0 2024-03-18T01:11:20.329 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:19.433+0000 7f6de37fe700 1 --2- 172.21.15.29:0/1686181815 >> [v2:172.21.15.29:6800/1702596206,v1:172.21.15.29:6801/1702596206] conn(0x7f6de4036740 0x7f6de4038bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:11:20.329 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:19.433+0000 7f6de37fe700 1 -- 172.21.15.29:0/1686181815 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f6dec04a7d0 con 0x7f6dfc1015b0 2024-03-18T01:11:20.329 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:19.433+0000 7f6dfa7fc700 1 --2- 172.21.15.29:0/1686181815 >> [v2:172.21.15.29:6800/1702596206,v1:172.21.15.29:6801/1702596206] conn(0x7f6de4036740 0x7f6de4038bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:11:20.329 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:19.434+0000 7f6dfa7fc700 1 --2- 172.21.15.29:0/1686181815 >> [v2:172.21.15.29:6800/1702596206,v1:172.21.15.29:6801/1702596206] conn(0x7f6de4036740 0x7f6de4038bf0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f6df0009990 tx=0x7f6df0006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:11:20.329 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:19.438+0000 7f6de37fe700 1 -- 172.21.15.29:0/1686181815 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175520 (secure 0 0 0) 0x7f6dec019670 con 0x7f6dfc1015b0 2024-03-18T01:11:20.329 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:19.684+0000 7f6e01261700 1 -- 172.21.15.29:0/1686181815 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) v1 -- 0x7f6dfc04bed0 con 0x7f6dfc1015b0 2024-03-18T01:11:20.329 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:19.737+0000 7f6de37fe700 1 -- 172.21.15.29:0/1686181815 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mgrmap(e 5) v1 ==== 43368+0+0 (secure 0 0 0) 0x7f6dec01a530 con 0x7f6dfc1015b0 2024-03-18T01:11:20.329 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:19.738+0000 7f6de37fe700 1 -- 172.21.15.29:0/1686181815 <== mon.0 v2:172.21.15.29:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) v1 ==== 86+0+0 (secure 0 0 0) 0x7f6dec022340 con 0x7f6dfc1015b0 2024-03-18T01:11:20.329 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:19.742+0000 7f6e01261700 1 -- 172.21.15.29:0/1686181815 >> [v2:172.21.15.29:6800/1702596206,v1:172.21.15.29:6801/1702596206] conn(0x7f6de4036740 msgr2=0x7f6de4038bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:11:20.329 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:19.742+0000 7f6e01261700 1 --2- 172.21.15.29:0/1686181815 >> [v2:172.21.15.29:6800/1702596206,v1:172.21.15.29:6801/1702596206] conn(0x7f6de4036740 0x7f6de4038bf0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f6df0009990 tx=0x7f6df0006e40 comp rx=0 tx=0).stop 2024-03-18T01:11:20.330 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:19.742+0000 7f6e01261700 1 -- 172.21.15.29:0/1686181815 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6dfc1015b0 msgr2=0x7f6dfc110490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:11:20.330 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:19.742+0000 7f6e01261700 1 --2- 172.21.15.29:0/1686181815 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6dfc1015b0 0x7f6dfc110490 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f6dec000c00 tx=0x7f6dec009250 comp rx=0 tx=0).stop 2024-03-18T01:11:20.330 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:19.742+0000 7f6e01261700 1 -- 172.21.15.29:0/1686181815 shutdown_connections 2024-03-18T01:11:20.330 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:19.742+0000 7f6e01261700 1 --2- 172.21.15.29:0/1686181815 >> [v2:172.21.15.29:6800/1702596206,v1:172.21.15.29:6801/1702596206] conn(0x7f6de4036740 0x7f6de4038bf0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:11:20.330 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:19.742+0000 7f6e01261700 1 --2- 172.21.15.29:0/1686181815 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6dfc1015b0 0x7f6dfc110490 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:11:20.330 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:19.742+0000 7f6e01261700 1 -- 172.21.15.29:0/1686181815 >> 172.21.15.29:0/1686181815 conn(0x7f6dfc0fcd70 msgr2=0x7f6dfc0fd8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:11:20.330 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:19.742+0000 7f6e01261700 1 -- 172.21.15.29:0/1686181815 shutdown_connections 2024-03-18T01:11:20.330 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:19.742+0000 7f6e01261700 1 -- 172.21.15.29:0/1686181815 wait complete. 2024-03-18T01:11:20.960 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:11:20 smithi029 ceph-mon[29331]: from='client.? 172.21.15.29:0/1686181815' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-03-18T01:11:20.960 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:11:20 smithi029 ceph-mon[29331]: mgrmap e5: smithi029.qviwhl(active, since 16s) 2024-03-18T01:11:21.617 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout { 2024-03-18T01:11:21.617 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-03-18T01:11:21.618 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-18T01:11:21.618 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "active_name": "smithi029.qviwhl", 2024-03-18T01:11:21.618 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-03-18T01:11:21.618 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout } 2024-03-18T01:11:21.618 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:20.754+0000 7f800c755700 1 Processor -- start 2024-03-18T01:11:21.618 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:20.754+0000 7f800c755700 1 -- start start 2024-03-18T01:11:21.618 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:20.755+0000 7f800c755700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8004102730 0x7f8004102b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:11:21.618 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:20.755+0000 7f800c755700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8004103080 con 0x7f8004102730 2024-03-18T01:11:21.618 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:20.755+0000 7f800a4f1700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8004102730 0x7f8004102b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:11:21.618 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:20.755+0000 7f800a4f1700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8004102730 0x7f8004102b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56396/0 (socket says 172.21.15.29:56396) 2024-03-18T01:11:21.618 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:20.755+0000 7f800a4f1700 1 -- 172.21.15.29:0/3202921223 learned_addr learned my addr 172.21.15.29:0/3202921223 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-18T01:11:21.618 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:20.755+0000 7f800a4f1700 1 -- 172.21.15.29:0/3202921223 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80041031c0 con 0x7f8004102730 2024-03-18T01:11:21.618 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:20.755+0000 7f800a4f1700 1 --2- 172.21.15.29:0/3202921223 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8004102730 0x7f8004102b40 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f80000098e0 tx=0x7f8000009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=93dc8865178d7f68 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:11:21.618 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:20.756+0000 7f80094ef700 1 -- 172.21.15.29:0/3202921223 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f800000b860 con 0x7f8004102730 2024-03-18T01:11:21.619 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:20.756+0000 7f80094ef700 1 -- 172.21.15.29:0/3202921223 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f800000be50 con 0x7f8004102730 2024-03-18T01:11:21.619 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:20.756+0000 7f80094ef700 1 -- 172.21.15.29:0/3202921223 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8000010380 con 0x7f8004102730 2024-03-18T01:11:21.619 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:20.756+0000 7f800c755700 1 -- 172.21.15.29:0/3202921223 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8004102730 msgr2=0x7f8004102b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:11:21.619 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:20.756+0000 7f800c755700 1 --2- 172.21.15.29:0/3202921223 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8004102730 0x7f8004102b40 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f80000098e0 tx=0x7f8000009250 comp rx=0 tx=0).stop 2024-03-18T01:11:21.619 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:20.756+0000 7f800c755700 1 -- 172.21.15.29:0/3202921223 shutdown_connections 2024-03-18T01:11:21.619 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:20.756+0000 7f800c755700 1 --2- 172.21.15.29:0/3202921223 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8004102730 0x7f8004102b40 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:11:21.619 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:20.756+0000 7f800c755700 1 -- 172.21.15.29:0/3202921223 >> 172.21.15.29:0/3202921223 conn(0x7f80040fdf10 msgr2=0x7f8004100320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:11:21.619 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:20.756+0000 7f800c755700 1 -- 172.21.15.29:0/3202921223 shutdown_connections 2024-03-18T01:11:21.619 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:20.756+0000 7f800c755700 1 -- 172.21.15.29:0/3202921223 wait complete. 2024-03-18T01:11:21.619 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:20.757+0000 7f800c755700 1 Processor -- start 2024-03-18T01:11:21.619 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:20.757+0000 7f800c755700 1 -- start start 2024-03-18T01:11:21.619 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:20.757+0000 7f800c755700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8004102730 0x7f8004078bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:11:21.619 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:20.757+0000 7f800c755700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8004079110 con 0x7f8004102730 2024-03-18T01:11:21.619 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:20.757+0000 7f800a4f1700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8004102730 0x7f8004078bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:11:21.619 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:20.757+0000 7f800a4f1700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8004102730 0x7f8004078bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56410/0 (socket says 172.21.15.29:56410) 2024-03-18T01:11:21.620 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:20.757+0000 7f800a4f1700 1 -- 172.21.15.29:0/341480452 learned_addr learned my addr 172.21.15.29:0/341480452 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-18T01:11:21.620 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:20.758+0000 7f800a4f1700 1 -- 172.21.15.29:0/341480452 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80000094f0 con 0x7f8004102730 2024-03-18T01:11:21.620 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:20.758+0000 7f800a4f1700 1 --2- 172.21.15.29:0/341480452 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8004102730 0x7f8004078bd0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f800000b440 tx=0x7f8000009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:11:21.620 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:20.758+0000 7f7ffaffd700 1 -- 172.21.15.29:0/341480452 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8000010760 con 0x7f8004102730 2024-03-18T01:11:21.620 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:20.758+0000 7f800c755700 1 -- 172.21.15.29:0/341480452 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8004077180 con 0x7f8004102730 2024-03-18T01:11:21.620 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:20.758+0000 7f7ffaffd700 1 -- 172.21.15.29:0/341480452 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f8000010d50 con 0x7f8004102730 2024-03-18T01:11:21.620 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:20.758+0000 7f7ffaffd700 1 -- 172.21.15.29:0/341480452 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f800001adc0 con 0x7f8004102730 2024-03-18T01:11:21.620 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:20.758+0000 7f800c755700 1 -- 172.21.15.29:0/341480452 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8004077670 con 0x7f8004102730 2024-03-18T01:11:21.620 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:20.759+0000 7f7ffaffd700 1 -- 172.21.15.29:0/341480452 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 43368+0+0 (secure 0 0 0) 0x7f80000193c0 con 0x7f8004102730 2024-03-18T01:11:21.620 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:20.759+0000 7f7ffaffd700 1 --2- 172.21.15.29:0/341480452 >> [v2:172.21.15.29:6800/1702596206,v1:172.21.15.29:6801/1702596206] conn(0x7f7ff00367e0 0x7f7ff0038c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:11:21.620 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:20.759+0000 7f7ffaffd700 1 -- 172.21.15.29:0/341480452 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f8000049f90 con 0x7f8004102730 2024-03-18T01:11:21.620 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:20.759+0000 7f8009cf0700 1 -- 172.21.15.29:0/341480452 >> [v2:172.21.15.29:6800/1702596206,v1:172.21.15.29:6801/1702596206] conn(0x7f7ff00367e0 msgr2=0x7f7ff0038c90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6800/1702596206 2024-03-18T01:11:21.620 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:20.759+0000 7f8009cf0700 1 --2- 172.21.15.29:0/341480452 >> [v2:172.21.15.29:6800/1702596206,v1:172.21.15.29:6801/1702596206] conn(0x7f7ff00367e0 0x7f7ff0038c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T01:11:21.620 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:20.759+0000 7f800c755700 1 -- 172.21.15.29:0/341480452 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f800404bed0 con 0x7f8004102730 2024-03-18T01:11:21.620 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:20.768+0000 7f7ffaffd700 1 -- 172.21.15.29:0/341480452 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175520 (secure 0 0 0) 0x7f8000023940 con 0x7f8004102730 2024-03-18T01:11:21.621 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:20.960+0000 7f8009cf0700 1 -- 172.21.15.29:0/341480452 >> [v2:172.21.15.29:6800/1702596206,v1:172.21.15.29:6801/1702596206] conn(0x7f7ff00367e0 msgr2=0x7f7ff0038c90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6800/1702596206 2024-03-18T01:11:21.621 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:20.960+0000 7f8009cf0700 1 --2- 172.21.15.29:0/341480452 >> [v2:172.21.15.29:6800/1702596206,v1:172.21.15.29:6801/1702596206] conn(0x7f7ff00367e0 0x7f7ff0038c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T01:11:21.621 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:21.013+0000 7f800c755700 1 -- 172.21.15.29:0/341480452 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7f8004077b20 con 0x7f8004102730 2024-03-18T01:11:21.621 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:21.013+0000 7f7ffaffd700 1 -- 172.21.15.29:0/341480452 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) v1 ==== 56+0+103 (secure 0 0 0) 0x7f8000023350 con 0x7f8004102730 2024-03-18T01:11:21.621 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:21.016+0000 7f800c755700 1 -- 172.21.15.29:0/341480452 >> [v2:172.21.15.29:6800/1702596206,v1:172.21.15.29:6801/1702596206] conn(0x7f7ff00367e0 msgr2=0x7f7ff0038c90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T01:11:21.621 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:21.016+0000 7f800c755700 1 --2- 172.21.15.29:0/341480452 >> [v2:172.21.15.29:6800/1702596206,v1:172.21.15.29:6801/1702596206] conn(0x7f7ff00367e0 0x7f7ff0038c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:11:21.621 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:21.016+0000 7f800c755700 1 -- 172.21.15.29:0/341480452 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8004102730 msgr2=0x7f8004078bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:11:21.621 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:21.016+0000 7f800c755700 1 --2- 172.21.15.29:0/341480452 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8004102730 0x7f8004078bd0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f800000b440 tx=0x7f8000009250 comp rx=0 tx=0).stop 2024-03-18T01:11:21.621 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:21.016+0000 7f800c755700 1 -- 172.21.15.29:0/341480452 shutdown_connections 2024-03-18T01:11:21.621 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:21.016+0000 7f800c755700 1 --2- 172.21.15.29:0/341480452 >> [v2:172.21.15.29:6800/1702596206,v1:172.21.15.29:6801/1702596206] conn(0x7f7ff00367e0 0x7f7ff0038c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:11:21.621 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:21.016+0000 7f800c755700 1 --2- 172.21.15.29:0/341480452 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8004102730 0x7f8004078bd0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:11:21.621 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:21.016+0000 7f800c755700 1 -- 172.21.15.29:0/341480452 >> 172.21.15.29:0/341480452 conn(0x7f80040fdf10 msgr2=0x7f80040feb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:11:21.621 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:21.016+0000 7f800c755700 1 -- 172.21.15.29:0/341480452 shutdown_connections 2024-03-18T01:11:21.621 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:21.016+0000 7f800c755700 1 -- 172.21.15.29:0/341480452 wait complete. 2024-03-18T01:11:21.621 INFO:teuthology.orchestra.run.smithi029.stdout:Waiting for the mgr to restart... 2024-03-18T01:11:21.622 INFO:teuthology.orchestra.run.smithi029.stdout:Waiting for mgr epoch 5... 2024-03-18T01:11:21.889 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:11:21 smithi029 ceph-mon[29331]: from='client.? 172.21.15.29:0/341480452' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-18T01:11:26.819 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:11:26 smithi029 ceph-mon[29331]: Active manager daemon smithi029.qviwhl restarted 2024-03-18T01:11:26.819 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:11:26 smithi029 ceph-mon[29331]: Activating manager daemon smithi029.qviwhl 2024-03-18T01:11:26.819 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:11:26 smithi029 ceph-mon[29331]: osdmap e2: 0 total, 0 up, 0 in 2024-03-18T01:11:26.819 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:11:26 smithi029 ceph-mon[29331]: mgrmap e6: smithi029.qviwhl(active, starting, since 0.0031988s) 2024-03-18T01:11:26.819 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:11:26 smithi029 ceph-mon[29331]: from='mgr.14116 172.21.15.29:0/340264435' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "mon metadata", "id": "smithi029"}]: dispatch 2024-03-18T01:11:26.819 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:11:26 smithi029 ceph-mon[29331]: from='mgr.14116 172.21.15.29:0/340264435' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "mgr metadata", "who": "smithi029.qviwhl", "id": "smithi029.qviwhl"}]: dispatch 2024-03-18T01:11:26.819 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:11:26 smithi029 ceph-mon[29331]: from='mgr.14116 172.21.15.29:0/340264435' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-18T01:11:26.819 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:11:26 smithi029 ceph-mon[29331]: from='mgr.14116 172.21.15.29:0/340264435' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-18T01:11:26.819 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:11:26 smithi029 ceph-mon[29331]: from='mgr.14116 172.21.15.29:0/340264435' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-18T01:11:26.820 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:11:26 smithi029 ceph-mon[29331]: Manager daemon smithi029.qviwhl is now available 2024-03-18T01:11:26.820 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:11:26 smithi029 ceph-mon[29331]: from='mgr.14116 172.21.15.29:0/340264435' entity='mgr.smithi029.qviwhl' 2024-03-18T01:11:26.820 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:11:26 smithi029 ceph-mon[29331]: from='mgr.14116 172.21.15.29:0/340264435' entity='mgr.smithi029.qviwhl' 2024-03-18T01:11:26.820 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:11:26 smithi029 ceph-mon[29331]: from='mgr.14116 172.21.15.29:0/340264435' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T01:11:26.820 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:11:26 smithi029 ceph-mon[29331]: from='mgr.14116 172.21.15.29:0/340264435' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T01:11:26.820 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:11:26 smithi029 ceph-mon[29331]: from='mgr.14116 172.21.15.29:0/340264435' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T01:11:27.819 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:11:27 smithi029 ceph-mon[29331]: Found migration_current of "None". Setting to last migration. 2024-03-18T01:11:27.819 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:11:27 smithi029 ceph-mon[29331]: from='mgr.14116 172.21.15.29:0/340264435' entity='mgr.smithi029.qviwhl' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi029.qviwhl/mirror_snapshot_schedule"}]: dispatch 2024-03-18T01:11:27.819 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:11:27 smithi029 ceph-mon[29331]: from='mgr.14116 172.21.15.29:0/340264435' entity='mgr.smithi029.qviwhl' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi029.qviwhl/trash_purge_schedule"}]: dispatch 2024-03-18T01:11:27.819 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:11:27 smithi029 ceph-mon[29331]: from='mgr.14116 172.21.15.29:0/340264435' entity='mgr.smithi029.qviwhl' 2024-03-18T01:11:27.819 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:11:27 smithi029 ceph-mon[29331]: from='mgr.14116 172.21.15.29:0/340264435' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T01:11:27.819 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:11:27 smithi029 ceph-mon[29331]: mgrmap e7: smithi029.qviwhl(active, since 1.00519s) 2024-03-18T01:11:28.683 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:11:28 smithi029 ceph-mon[29331]: [18/Mar/2024:01:11:27] ENGINE Bus STARTING 2024-03-18T01:11:28.683 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:11:28 smithi029 ceph-mon[29331]: [18/Mar/2024:01:11:27] ENGINE Serving on https://172.21.15.29:7150 2024-03-18T01:11:28.683 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:11:28 smithi029 ceph-mon[29331]: [18/Mar/2024:01:11:27] ENGINE Bus STARTED 2024-03-18T01:11:28.683 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:11:28 smithi029 ceph-mon[29331]: from='client.14120 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-03-18T01:11:28.683 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:11:28 smithi029 ceph-mon[29331]: from='client.14120 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-03-18T01:11:28.693 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout { 2024-03-18T01:11:28.693 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-03-18T01:11:28.693 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "initialized": true 2024-03-18T01:11:28.694 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout } 2024-03-18T01:11:28.694 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:22.037+0000 7ff65d172700 1 Processor -- start 2024-03-18T01:11:28.694 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:22.037+0000 7ff65d172700 1 -- start start 2024-03-18T01:11:28.694 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:22.038+0000 7ff65d172700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff658077bc0 0x7ff6580760d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:11:28.694 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:22.038+0000 7ff65d172700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff658077fd0 con 0x7ff658077bc0 2024-03-18T01:11:28.694 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:22.038+0000 7ff656d9d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff658077bc0 0x7ff6580760d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:11:28.694 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:22.038+0000 7ff656d9d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff658077bc0 0x7ff6580760d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56412/0 (socket says 172.21.15.29:56412) 2024-03-18T01:11:28.694 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:22.038+0000 7ff656d9d700 1 -- 172.21.15.29:0/633314546 learned_addr learned my addr 172.21.15.29:0/633314546 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-18T01:11:28.694 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:22.038+0000 7ff656d9d700 1 -- 172.21.15.29:0/633314546 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff658076610 con 0x7ff658077bc0 2024-03-18T01:11:28.694 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:22.038+0000 7ff656d9d700 1 --2- 172.21.15.29:0/633314546 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff658077bc0 0x7ff6580760d0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7ff6480098e0 tx=0x7ff648009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=346666618bfe35bb server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:11:28.694 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:22.038+0000 7ff65659c700 1 -- 172.21.15.29:0/633314546 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff64800b860 con 0x7ff658077bc0 2024-03-18T01:11:28.694 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:22.038+0000 7ff65659c700 1 -- 172.21.15.29:0/633314546 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7ff64800be50 con 0x7ff658077bc0 2024-03-18T01:11:28.694 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:22.038+0000 7ff65659c700 1 -- 172.21.15.29:0/633314546 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff648010380 con 0x7ff658077bc0 2024-03-18T01:11:28.694 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:22.039+0000 7ff65d172700 1 -- 172.21.15.29:0/633314546 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff658077bc0 msgr2=0x7ff6580760d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:11:28.695 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:22.039+0000 7ff65d172700 1 --2- 172.21.15.29:0/633314546 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff658077bc0 0x7ff6580760d0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7ff6480098e0 tx=0x7ff648009250 comp rx=0 tx=0).stop 2024-03-18T01:11:28.695 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:22.039+0000 7ff65d172700 1 -- 172.21.15.29:0/633314546 shutdown_connections 2024-03-18T01:11:28.695 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:22.039+0000 7ff65d172700 1 --2- 172.21.15.29:0/633314546 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff658077bc0 0x7ff6580760d0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:11:28.695 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:22.039+0000 7ff65d172700 1 -- 172.21.15.29:0/633314546 >> 172.21.15.29:0/633314546 conn(0x7ff6580fde70 msgr2=0x7ff658100280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:11:28.695 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:22.039+0000 7ff65d172700 1 -- 172.21.15.29:0/633314546 shutdown_connections 2024-03-18T01:11:28.695 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:22.039+0000 7ff65d172700 1 -- 172.21.15.29:0/633314546 wait complete. 2024-03-18T01:11:28.695 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:22.039+0000 7ff65d172700 1 Processor -- start 2024-03-18T01:11:28.695 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:22.040+0000 7ff65d172700 1 -- start start 2024-03-18T01:11:28.695 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:22.040+0000 7ff65d172700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff658077bc0 0x7ff6581137c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:11:28.695 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:22.040+0000 7ff65d172700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff658113d00 con 0x7ff658077bc0 2024-03-18T01:11:28.695 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:22.040+0000 7ff656d9d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff658077bc0 0x7ff6581137c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:11:28.695 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:22.040+0000 7ff656d9d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff658077bc0 0x7ff6581137c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56424/0 (socket says 172.21.15.29:56424) 2024-03-18T01:11:28.695 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:22.040+0000 7ff656d9d700 1 -- 172.21.15.29:0/53854544 learned_addr learned my addr 172.21.15.29:0/53854544 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-18T01:11:28.695 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:22.040+0000 7ff656d9d700 1 -- 172.21.15.29:0/53854544 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6480094f0 con 0x7ff658077bc0 2024-03-18T01:11:28.695 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:22.040+0000 7ff656d9d700 1 --2- 172.21.15.29:0/53854544 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff658077bc0 0x7ff6581137c0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7ff648006fd0 tx=0x7ff648009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:11:28.696 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:22.041+0000 7ff64ffff700 1 -- 172.21.15.29:0/53854544 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff648010760 con 0x7ff658077bc0 2024-03-18T01:11:28.696 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:22.041+0000 7ff64ffff700 1 -- 172.21.15.29:0/53854544 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7ff648010d50 con 0x7ff658077bc0 2024-03-18T01:11:28.696 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:22.041+0000 7ff65d172700 1 -- 172.21.15.29:0/53854544 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff658113f00 con 0x7ff658077bc0 2024-03-18T01:11:28.696 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:22.041+0000 7ff64ffff700 1 -- 172.21.15.29:0/53854544 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff64801ae10 con 0x7ff658077bc0 2024-03-18T01:11:28.696 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:22.041+0000 7ff65d172700 1 -- 172.21.15.29:0/53854544 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff658114a70 con 0x7ff658077bc0 2024-03-18T01:11:28.696 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:22.042+0000 7ff64ffff700 1 -- 172.21.15.29:0/53854544 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 43368+0+0 (secure 0 0 0) 0x7ff6480193c0 con 0x7ff658077bc0 2024-03-18T01:11:28.696 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:22.042+0000 7ff64ffff700 1 --2- 172.21.15.29:0/53854544 >> [v2:172.21.15.29:6800/1702596206,v1:172.21.15.29:6801/1702596206] conn(0x7ff644036790 0x7ff644038c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:11:28.696 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:22.042+0000 7ff65d172700 1 -- 172.21.15.29:0/53854544 --> [v2:172.21.15.29:6800/1702596206,v1:172.21.15.29:6801/1702596206] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff65805f5f0 con 0x7ff644036790 2024-03-18T01:11:28.696 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:22.042+0000 7ff64ffff700 1 -- 172.21.15.29:0/53854544 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7ff648049f60 con 0x7ff658077bc0 2024-03-18T01:11:28.696 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:22.042+0000 7ff64e59c700 1 -- 172.21.15.29:0/53854544 >> [v2:172.21.15.29:6800/1702596206,v1:172.21.15.29:6801/1702596206] conn(0x7ff644036790 msgr2=0x7ff644038c40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6800/1702596206 2024-03-18T01:11:28.696 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:22.042+0000 7ff64e59c700 1 --2- 172.21.15.29:0/53854544 >> [v2:172.21.15.29:6800/1702596206,v1:172.21.15.29:6801/1702596206] conn(0x7ff644036790 0x7ff644038c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T01:11:28.696 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:22.243+0000 7ff64e59c700 1 -- 172.21.15.29:0/53854544 >> [v2:172.21.15.29:6800/1702596206,v1:172.21.15.29:6801/1702596206] conn(0x7ff644036790 msgr2=0x7ff644038c40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6800/1702596206 2024-03-18T01:11:28.696 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:22.243+0000 7ff64e59c700 1 --2- 172.21.15.29:0/53854544 >> [v2:172.21.15.29:6800/1702596206,v1:172.21.15.29:6801/1702596206] conn(0x7ff644036790 0x7ff644038c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T01:11:28.696 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:22.644+0000 7ff64e59c700 1 -- 172.21.15.29:0/53854544 >> [v2:172.21.15.29:6800/1702596206,v1:172.21.15.29:6801/1702596206] conn(0x7ff644036790 msgr2=0x7ff644038c40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6800/1702596206 2024-03-18T01:11:28.696 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:22.644+0000 7ff64e59c700 1 --2- 172.21.15.29:0/53854544 >> [v2:172.21.15.29:6800/1702596206,v1:172.21.15.29:6801/1702596206] conn(0x7ff644036790 0x7ff644038c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-18T01:11:28.697 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:23.445+0000 7ff64e59c700 1 -- 172.21.15.29:0/53854544 >> [v2:172.21.15.29:6800/1702596206,v1:172.21.15.29:6801/1702596206] conn(0x7ff644036790 msgr2=0x7ff644038c40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6800/1702596206 2024-03-18T01:11:28.697 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:23.445+0000 7ff64e59c700 1 --2- 172.21.15.29:0/53854544 >> [v2:172.21.15.29:6800/1702596206,v1:172.21.15.29:6801/1702596206] conn(0x7ff644036790 0x7ff644038c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-18T01:11:28.697 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:25.046+0000 7ff64e59c700 1 -- 172.21.15.29:0/53854544 >> [v2:172.21.15.29:6800/1702596206,v1:172.21.15.29:6801/1702596206] conn(0x7ff644036790 msgr2=0x7ff644038c40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6800/1702596206 2024-03-18T01:11:28.697 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:25.046+0000 7ff64e59c700 1 --2- 172.21.15.29:0/53854544 >> [v2:172.21.15.29:6800/1702596206,v1:172.21.15.29:6801/1702596206] conn(0x7ff644036790 0x7ff644038c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-03-18T01:11:28.697 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:26.434+0000 7ff64ffff700 1 -- 172.21.15.29:0/53854544 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mgrmap(e 6) v1 ==== 43135+0+0 (secure 0 0 0) 0x7ff64802c350 con 0x7ff658077bc0 2024-03-18T01:11:28.697 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:26.435+0000 7ff64ffff700 1 -- 172.21.15.29:0/53854544 >> [v2:172.21.15.29:6800/1702596206,v1:172.21.15.29:6801/1702596206] conn(0x7ff644036790 msgr2=0x7ff644038c40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T01:11:28.697 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:26.435+0000 7ff64ffff700 1 --2- 172.21.15.29:0/53854544 >> [v2:172.21.15.29:6800/1702596206,v1:172.21.15.29:6801/1702596206] conn(0x7ff644036790 0x7ff644038c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:11:28.697 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:27.436+0000 7ff64ffff700 1 -- 172.21.15.29:0/53854544 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mgrmap(e 7) v1 ==== 43262+0+0 (secure 0 0 0) 0x7ff64802ad50 con 0x7ff658077bc0 2024-03-18T01:11:28.697 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:27.436+0000 7ff64ffff700 1 --2- 172.21.15.29:0/53854544 >> [v2:172.21.15.29:6800/950856841,v1:172.21.15.29:6801/950856841] conn(0x7ff644039f50 0x7ff64403c330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:11:28.697 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:27.436+0000 7ff64ffff700 1 -- 172.21.15.29:0/53854544 --> [v2:172.21.15.29:6800/950856841,v1:172.21.15.29:6801/950856841] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff65805f5f0 con 0x7ff644039f50 2024-03-18T01:11:28.697 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:27.436+0000 7ff64e59c700 1 --2- 172.21.15.29:0/53854544 >> [v2:172.21.15.29:6800/950856841,v1:172.21.15.29:6801/950856841] conn(0x7ff644039f50 0x7ff64403c330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:11:28.697 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:27.437+0000 7ff64e59c700 1 --2- 172.21.15.29:0/53854544 >> [v2:172.21.15.29:6800/950856841,v1:172.21.15.29:6801/950856841] conn(0x7ff644039f50 0x7ff64403c330 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7ff6400039d0 tx=0x7ff640009350 comp rx=0 tx=0).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:11:28.697 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:27.437+0000 7ff64ffff700 1 -- 172.21.15.29:0/53854544 <== mgr.14116 v2:172.21.15.29:6800/950856841 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6549 (secure 0 0 0) 0x7ff65805f5f0 con 0x7ff644039f50 2024-03-18T01:11:28.697 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:27.444+0000 7ff65d172700 1 -- 172.21.15.29:0/53854544 --> [v2:172.21.15.29:6800/950856841,v1:172.21.15.29:6801/950856841] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7ff658114390 con 0x7ff644039f50 2024-03-18T01:11:28.698 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:27.444+0000 7ff64ffff700 1 -- 172.21.15.29:0/53854544 <== mgr.14116 v2:172.21.15.29:6800/950856841 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7ff658114390 con 0x7ff644039f50 2024-03-18T01:11:28.698 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:27.444+0000 7ff65d172700 1 -- 172.21.15.29:0/53854544 >> [v2:172.21.15.29:6800/950856841,v1:172.21.15.29:6801/950856841] conn(0x7ff644039f50 msgr2=0x7ff64403c330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:11:28.698 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:27.444+0000 7ff65d172700 1 --2- 172.21.15.29:0/53854544 >> [v2:172.21.15.29:6800/950856841,v1:172.21.15.29:6801/950856841] conn(0x7ff644039f50 0x7ff64403c330 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7ff6400039d0 tx=0x7ff640009350 comp rx=0 tx=0).stop 2024-03-18T01:11:28.698 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:27.444+0000 7ff65d172700 1 -- 172.21.15.29:0/53854544 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff658077bc0 msgr2=0x7ff6581137c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:11:28.698 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:27.444+0000 7ff65d172700 1 --2- 172.21.15.29:0/53854544 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff658077bc0 0x7ff6581137c0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7ff648006fd0 tx=0x7ff648009250 comp rx=0 tx=0).stop 2024-03-18T01:11:28.698 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:27.444+0000 7ff65d172700 1 -- 172.21.15.29:0/53854544 shutdown_connections 2024-03-18T01:11:28.698 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:27.444+0000 7ff65d172700 1 --2- 172.21.15.29:0/53854544 >> [v2:172.21.15.29:6800/950856841,v1:172.21.15.29:6801/950856841] conn(0x7ff644039f50 0x7ff64403c330 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:11:28.698 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:27.444+0000 7ff65d172700 1 --2- 172.21.15.29:0/53854544 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff658077bc0 0x7ff6581137c0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:11:28.698 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:27.444+0000 7ff65d172700 1 --2- 172.21.15.29:0/53854544 >> [v2:172.21.15.29:6800/1702596206,v1:172.21.15.29:6801/1702596206] conn(0x7ff644036790 0x7ff644038c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:11:28.698 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:27.444+0000 7ff65d172700 1 -- 172.21.15.29:0/53854544 >> 172.21.15.29:0/53854544 conn(0x7ff6580fde70 msgr2=0x7ff6580fea70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:11:28.698 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:27.445+0000 7ff65d172700 1 -- 172.21.15.29:0/53854544 shutdown_connections 2024-03-18T01:11:28.698 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:27.445+0000 7ff65d172700 1 -- 172.21.15.29:0/53854544 wait complete. 2024-03-18T01:11:28.698 INFO:teuthology.orchestra.run.smithi029.stdout:mgr epoch 5 is available 2024-03-18T01:11:28.698 INFO:teuthology.orchestra.run.smithi029.stdout:Setting orchestrator backend to cephadm... 2024-03-18T01:11:29.636 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:11:29 smithi029 ceph-mon[29331]: mgrmap e8: smithi029.qviwhl(active, since 2s) 2024-03-18T01:11:29.636 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:11:29 smithi029 ceph-mon[29331]: from='mgr.14116 172.21.15.29:0/340264435' entity='mgr.smithi029.qviwhl' 2024-03-18T01:11:29.636 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:11:29 smithi029 ceph-mon[29331]: from='mgr.14116 172.21.15.29:0/340264435' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T01:11:29.914 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:29.115+0000 7faa082c1700 1 Processor -- start 2024-03-18T01:11:29.914 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:29.115+0000 7faa082c1700 1 -- start start 2024-03-18T01:11:29.914 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:29.115+0000 7faa082c1700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faa00102740 0x7faa00102b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:11:29.914 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:29.115+0000 7faa082c1700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa00103090 con 0x7faa00102740 2024-03-18T01:11:29.914 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:29.115+0000 7faa0605d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faa00102740 0x7faa00102b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:11:29.915 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:29.115+0000 7faa0605d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faa00102740 0x7faa00102b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53530/0 (socket says 172.21.15.29:53530) 2024-03-18T01:11:29.915 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:29.115+0000 7faa0605d700 1 -- 172.21.15.29:0/933338245 learned_addr learned my addr 172.21.15.29:0/933338245 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-18T01:11:29.915 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:29.116+0000 7faa0605d700 1 -- 172.21.15.29:0/933338245 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa001031d0 con 0x7faa00102740 2024-03-18T01:11:29.915 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:29.116+0000 7faa0605d700 1 --2- 172.21.15.29:0/933338245 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faa00102740 0x7faa00102b50 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fa9fc0098e0 tx=0x7fa9fc009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af8a9477280a90a6 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:11:29.915 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:29.116+0000 7faa0505b700 1 -- 172.21.15.29:0/933338245 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa9fc00b860 con 0x7faa00102740 2024-03-18T01:11:29.915 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:29.116+0000 7faa0505b700 1 -- 172.21.15.29:0/933338245 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7fa9fc00be50 con 0x7faa00102740 2024-03-18T01:11:29.915 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:29.116+0000 7faa0505b700 1 -- 172.21.15.29:0/933338245 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa9fc010380 con 0x7faa00102740 2024-03-18T01:11:29.915 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:29.116+0000 7faa082c1700 1 -- 172.21.15.29:0/933338245 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faa00102740 msgr2=0x7faa00102b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:11:29.915 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:29.116+0000 7faa082c1700 1 --2- 172.21.15.29:0/933338245 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faa00102740 0x7faa00102b50 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fa9fc0098e0 tx=0x7fa9fc009250 comp rx=0 tx=0).stop 2024-03-18T01:11:29.915 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:29.117+0000 7faa082c1700 1 -- 172.21.15.29:0/933338245 shutdown_connections 2024-03-18T01:11:29.915 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:29.117+0000 7faa082c1700 1 --2- 172.21.15.29:0/933338245 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faa00102740 0x7faa00102b50 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:11:29.915 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:29.117+0000 7faa082c1700 1 -- 172.21.15.29:0/933338245 >> 172.21.15.29:0/933338245 conn(0x7faa000fdea0 msgr2=0x7faa001002d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:11:29.915 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:29.117+0000 7faa082c1700 1 -- 172.21.15.29:0/933338245 shutdown_connections 2024-03-18T01:11:29.915 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:29.117+0000 7faa082c1700 1 -- 172.21.15.29:0/933338245 wait complete. 2024-03-18T01:11:29.915 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:29.117+0000 7faa082c1700 1 Processor -- start 2024-03-18T01:11:29.915 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:29.117+0000 7faa082c1700 1 -- start start 2024-03-18T01:11:29.916 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:29.117+0000 7faa082c1700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faa00102740 0x7faa001115b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:11:29.916 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:29.117+0000 7faa082c1700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa00111af0 con 0x7faa00102740 2024-03-18T01:11:29.916 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:29.117+0000 7faa0605d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faa00102740 0x7faa001115b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:11:29.916 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:29.118+0000 7faa0605d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faa00102740 0x7faa001115b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53536/0 (socket says 172.21.15.29:53536) 2024-03-18T01:11:29.916 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:29.118+0000 7faa0605d700 1 -- 172.21.15.29:0/3761741428 learned_addr learned my addr 172.21.15.29:0/3761741428 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-18T01:11:29.916 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:29.118+0000 7faa0605d700 1 -- 172.21.15.29:0/3761741428 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9fc0094f0 con 0x7faa00102740 2024-03-18T01:11:29.916 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:29.118+0000 7faa0605d700 1 --2- 172.21.15.29:0/3761741428 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faa00102740 0x7faa001115b0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fa9fc006b00 tx=0x7fa9fc009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:11:29.916 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:29.118+0000 7fa9f6ffd700 1 -- 172.21.15.29:0/3761741428 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa9fc010760 con 0x7faa00102740 2024-03-18T01:11:29.916 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:29.118+0000 7fa9f6ffd700 1 -- 172.21.15.29:0/3761741428 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7fa9fc010d50 con 0x7faa00102740 2024-03-18T01:11:29.916 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:29.118+0000 7faa082c1700 1 -- 172.21.15.29:0/3761741428 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa00111cf0 con 0x7faa00102740 2024-03-18T01:11:29.916 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:29.118+0000 7fa9f6ffd700 1 -- 172.21.15.29:0/3761741428 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa9fc01ad40 con 0x7faa00102740 2024-03-18T01:11:29.916 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:29.118+0000 7faa082c1700 1 -- 172.21.15.29:0/3761741428 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa00112860 con 0x7faa00102740 2024-03-18T01:11:29.916 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:29.119+0000 7fa9f6ffd700 1 -- 172.21.15.29:0/3761741428 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7fa9fc0108c0 con 0x7faa00102740 2024-03-18T01:11:29.916 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:29.119+0000 7faa082c1700 1 -- 172.21.15.29:0/3761741428 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa0005f5f0 con 0x7faa00102740 2024-03-18T01:11:29.916 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:29.119+0000 7fa9f6ffd700 1 --2- 172.21.15.29:0/3761741428 >> [v2:172.21.15.29:6800/950856841,v1:172.21.15.29:6801/950856841] conn(0x7fa9ec0367e0 0x7fa9ec038c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:11:29.916 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:29.119+0000 7fa9f6ffd700 1 -- 172.21.15.29:0/3761741428 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7fa9fc04a580 con 0x7faa00102740 2024-03-18T01:11:29.917 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:29.119+0000 7faa0585c700 1 --2- 172.21.15.29:0/3761741428 >> [v2:172.21.15.29:6800/950856841,v1:172.21.15.29:6801/950856841] conn(0x7fa9ec0367e0 0x7fa9ec038c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:11:29.917 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:29.120+0000 7faa0585c700 1 --2- 172.21.15.29:0/3761741428 >> [v2:172.21.15.29:6800/950856841,v1:172.21.15.29:6801/950856841] conn(0x7fa9ec0367e0 0x7fa9ec038c90 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fa9f0009990 tx=0x7fa9f0006e40 comp rx=0 tx=0).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:11:29.917 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:29.124+0000 7fa9f6ffd700 1 -- 172.21.15.29:0/3761741428 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175520 (secure 0 0 0) 0x7fa9fc018b40 con 0x7faa00102740 2024-03-18T01:11:29.917 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:29.332+0000 7faa082c1700 1 -- 172.21.15.29:0/3761741428 --> [v2:172.21.15.29:6800/950856841,v1:172.21.15.29:6801/950856841] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) v1 -- 0x7faa000ff120 con 0x7fa9ec0367e0 2024-03-18T01:11:29.917 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:29.340+0000 7fa9f6ffd700 1 -- 172.21.15.29:0/3761741428 <== mgr.14116 v2:172.21.15.29:6800/950856841 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7faa000ff120 con 0x7fa9ec0367e0 2024-03-18T01:11:29.917 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:29.342+0000 7faa082c1700 1 -- 172.21.15.29:0/3761741428 >> [v2:172.21.15.29:6800/950856841,v1:172.21.15.29:6801/950856841] conn(0x7fa9ec0367e0 msgr2=0x7fa9ec038c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:11:29.917 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:29.342+0000 7faa082c1700 1 --2- 172.21.15.29:0/3761741428 >> [v2:172.21.15.29:6800/950856841,v1:172.21.15.29:6801/950856841] conn(0x7fa9ec0367e0 0x7fa9ec038c90 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fa9f0009990 tx=0x7fa9f0006e40 comp rx=0 tx=0).stop 2024-03-18T01:11:29.917 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:29.342+0000 7faa082c1700 1 -- 172.21.15.29:0/3761741428 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faa00102740 msgr2=0x7faa001115b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:11:29.917 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:29.342+0000 7faa082c1700 1 --2- 172.21.15.29:0/3761741428 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faa00102740 0x7faa001115b0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fa9fc006b00 tx=0x7fa9fc009250 comp rx=0 tx=0).stop 2024-03-18T01:11:29.917 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:29.342+0000 7faa082c1700 1 -- 172.21.15.29:0/3761741428 shutdown_connections 2024-03-18T01:11:29.917 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:29.342+0000 7faa082c1700 1 --2- 172.21.15.29:0/3761741428 >> [v2:172.21.15.29:6800/950856841,v1:172.21.15.29:6801/950856841] conn(0x7fa9ec0367e0 0x7fa9ec038c90 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:11:29.917 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:29.343+0000 7faa082c1700 1 --2- 172.21.15.29:0/3761741428 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faa00102740 0x7faa001115b0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:11:29.917 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:29.343+0000 7faa082c1700 1 -- 172.21.15.29:0/3761741428 >> 172.21.15.29:0/3761741428 conn(0x7faa000fdea0 msgr2=0x7faa000fea10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:11:29.917 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:29.343+0000 7faa082c1700 1 -- 172.21.15.29:0/3761741428 shutdown_connections 2024-03-18T01:11:29.917 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:29.343+0000 7faa082c1700 1 -- 172.21.15.29:0/3761741428 wait complete. 2024-03-18T01:11:30.551 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:11:30 smithi029 ceph-mon[29331]: from='client.14128 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T01:11:31.111 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout value unchanged 2024-03-18T01:11:31.112 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:30.347+0000 7f5be55e7700 1 Processor -- start 2024-03-18T01:11:31.112 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:30.347+0000 7f5be55e7700 1 -- start start 2024-03-18T01:11:31.112 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:30.347+0000 7f5be55e7700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5be0078c60 0x7f5be00771c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:11:31.112 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:30.347+0000 7f5be55e7700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5be0079070 con 0x7f5be0078c60 2024-03-18T01:11:31.112 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:30.347+0000 7f5bdeffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5be0078c60 0x7f5be00771c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:11:31.112 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:30.347+0000 7f5bdeffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5be0078c60 0x7f5be00771c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53542/0 (socket says 172.21.15.29:53542) 2024-03-18T01:11:31.112 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:30.347+0000 7f5bdeffd700 1 -- 172.21.15.29:0/3914449341 learned_addr learned my addr 172.21.15.29:0/3914449341 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-18T01:11:31.112 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:30.348+0000 7f5bdeffd700 1 -- 172.21.15.29:0/3914449341 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5be0077700 con 0x7f5be0078c60 2024-03-18T01:11:31.112 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:30.348+0000 7f5bdeffd700 1 --2- 172.21.15.29:0/3914449341 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5be0078c60 0x7f5be00771c0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f5bd00098e0 tx=0x7f5bd0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c662bb5bde88c75 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:11:31.112 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:30.348+0000 7f5bddffb700 1 -- 172.21.15.29:0/3914449341 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5bd000b860 con 0x7f5be0078c60 2024-03-18T01:11:31.112 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:30.348+0000 7f5bddffb700 1 -- 172.21.15.29:0/3914449341 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f5bd000be50 con 0x7f5be0078c60 2024-03-18T01:11:31.112 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:30.348+0000 7f5bddffb700 1 -- 172.21.15.29:0/3914449341 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5bd0010380 con 0x7f5be0078c60 2024-03-18T01:11:31.112 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:30.348+0000 7f5be55e7700 1 -- 172.21.15.29:0/3914449341 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5be0078c60 msgr2=0x7f5be00771c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:11:31.113 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:30.348+0000 7f5be55e7700 1 --2- 172.21.15.29:0/3914449341 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5be0078c60 0x7f5be00771c0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f5bd00098e0 tx=0x7f5bd0009250 comp rx=0 tx=0).stop 2024-03-18T01:11:31.113 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:30.348+0000 7f5be55e7700 1 -- 172.21.15.29:0/3914449341 shutdown_connections 2024-03-18T01:11:31.113 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:30.348+0000 7f5be55e7700 1 --2- 172.21.15.29:0/3914449341 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5be0078c60 0x7f5be00771c0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:11:31.113 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:30.348+0000 7f5be55e7700 1 -- 172.21.15.29:0/3914449341 >> 172.21.15.29:0/3914449341 conn(0x7f5be00fdea0 msgr2=0x7f5be01002b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:11:31.113 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:30.349+0000 7f5be55e7700 1 -- 172.21.15.29:0/3914449341 shutdown_connections 2024-03-18T01:11:31.113 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:30.349+0000 7f5be55e7700 1 -- 172.21.15.29:0/3914449341 wait complete. 2024-03-18T01:11:31.113 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:30.349+0000 7f5be55e7700 1 Processor -- start 2024-03-18T01:11:31.113 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:30.349+0000 7f5be55e7700 1 -- start start 2024-03-18T01:11:31.113 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:30.349+0000 7f5be55e7700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5be0078c60 0x7f5be0113800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:11:31.113 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:30.349+0000 7f5be55e7700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5be0113d40 con 0x7f5be0078c60 2024-03-18T01:11:31.113 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:30.350+0000 7f5bdeffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5be0078c60 0x7f5be0113800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:11:31.113 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:30.350+0000 7f5bdeffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5be0078c60 0x7f5be0113800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53558/0 (socket says 172.21.15.29:53558) 2024-03-18T01:11:31.113 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:30.350+0000 7f5bdeffd700 1 -- 172.21.15.29:0/2600134442 learned_addr learned my addr 172.21.15.29:0/2600134442 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-18T01:11:31.113 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:30.350+0000 7f5bdeffd700 1 -- 172.21.15.29:0/2600134442 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bd00094f0 con 0x7f5be0078c60 2024-03-18T01:11:31.113 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:30.350+0000 7f5bdeffd700 1 --2- 172.21.15.29:0/2600134442 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5be0078c60 0x7f5be0113800 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f5bd0006fd0 tx=0x7f5bd0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:11:31.113 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:30.350+0000 7f5bc77fe700 1 -- 172.21.15.29:0/2600134442 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5bd00107c0 con 0x7f5be0078c60 2024-03-18T01:11:31.114 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:30.350+0000 7f5be55e7700 1 -- 172.21.15.29:0/2600134442 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5be0113f40 con 0x7f5be0078c60 2024-03-18T01:11:31.114 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:30.350+0000 7f5bc77fe700 1 -- 172.21.15.29:0/2600134442 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f5bd0010db0 con 0x7f5be0078c60 2024-03-18T01:11:31.114 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:30.350+0000 7f5bc77fe700 1 -- 172.21.15.29:0/2600134442 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5bd001ada0 con 0x7f5be0078c60 2024-03-18T01:11:31.114 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:30.350+0000 7f5be55e7700 1 -- 172.21.15.29:0/2600134442 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5be0114ab0 con 0x7f5be0078c60 2024-03-18T01:11:31.114 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:30.351+0000 7f5bc77fe700 1 -- 172.21.15.29:0/2600134442 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7f5bd0019450 con 0x7f5be0078c60 2024-03-18T01:11:31.114 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:30.351+0000 7f5bc77fe700 1 --2- 172.21.15.29:0/2600134442 >> [v2:172.21.15.29:6800/950856841,v1:172.21.15.29:6801/950856841] conn(0x7f5bc8036790 0x7f5bc8038c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:11:31.114 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:30.351+0000 7f5be55e7700 1 -- 172.21.15.29:0/2600134442 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5be005f5f0 con 0x7f5be0078c60 2024-03-18T01:11:31.114 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:30.351+0000 7f5bc77fe700 1 -- 172.21.15.29:0/2600134442 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f5bd0049ee0 con 0x7f5be0078c60 2024-03-18T01:11:31.114 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:30.352+0000 7f5bde7fc700 1 --2- 172.21.15.29:0/2600134442 >> [v2:172.21.15.29:6800/950856841,v1:172.21.15.29:6801/950856841] conn(0x7f5bc8036790 0x7f5bc8038c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:11:31.114 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:30.352+0000 7f5bde7fc700 1 --2- 172.21.15.29:0/2600134442 >> [v2:172.21.15.29:6800/950856841,v1:172.21.15.29:6801/950856841] conn(0x7f5bc8036790 0x7f5bc8038c40 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f5bd4009990 tx=0x7f5bd4006e40 comp rx=0 tx=0).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:11:31.114 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:30.356+0000 7f5bc77fe700 1 -- 172.21.15.29:0/2600134442 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175520 (secure 0 0 0) 0x7f5bd0029020 con 0x7f5be0078c60 2024-03-18T01:11:31.114 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:30.550+0000 7f5be55e7700 1 -- 172.21.15.29:0/2600134442 --> [v2:172.21.15.29:6800/950856841,v1:172.21.15.29:6801/950856841] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) v1 -- 0x7f5be00ff290 con 0x7f5bc8036790 2024-03-18T01:11:31.114 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:30.551+0000 7f5bc77fe700 1 -- 172.21.15.29:0/2600134442 <== mgr.14116 v2:172.21.15.29:6800/950856841 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+16 (secure 0 0 0) 0x7f5be00ff290 con 0x7f5bc8036790 2024-03-18T01:11:31.114 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:30.553+0000 7f5be55e7700 1 -- 172.21.15.29:0/2600134442 >> [v2:172.21.15.29:6800/950856841,v1:172.21.15.29:6801/950856841] conn(0x7f5bc8036790 msgr2=0x7f5bc8038c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:11:31.114 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:30.553+0000 7f5be55e7700 1 --2- 172.21.15.29:0/2600134442 >> [v2:172.21.15.29:6800/950856841,v1:172.21.15.29:6801/950856841] conn(0x7f5bc8036790 0x7f5bc8038c40 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f5bd4009990 tx=0x7f5bd4006e40 comp rx=0 tx=0).stop 2024-03-18T01:11:31.115 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:30.553+0000 7f5be55e7700 1 -- 172.21.15.29:0/2600134442 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5be0078c60 msgr2=0x7f5be0113800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:11:31.115 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:30.553+0000 7f5be55e7700 1 --2- 172.21.15.29:0/2600134442 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5be0078c60 0x7f5be0113800 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f5bd0006fd0 tx=0x7f5bd0009250 comp rx=0 tx=0).stop 2024-03-18T01:11:31.115 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:30.553+0000 7f5be55e7700 1 -- 172.21.15.29:0/2600134442 shutdown_connections 2024-03-18T01:11:31.115 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:30.553+0000 7f5be55e7700 1 --2- 172.21.15.29:0/2600134442 >> [v2:172.21.15.29:6800/950856841,v1:172.21.15.29:6801/950856841] conn(0x7f5bc8036790 0x7f5bc8038c40 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:11:31.115 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:30.553+0000 7f5be55e7700 1 --2- 172.21.15.29:0/2600134442 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5be0078c60 0x7f5be0113800 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:11:31.115 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:30.553+0000 7f5be55e7700 1 -- 172.21.15.29:0/2600134442 >> 172.21.15.29:0/2600134442 conn(0x7f5be00fdea0 msgr2=0x7f5be00feac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:11:31.115 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:30.553+0000 7f5be55e7700 1 -- 172.21.15.29:0/2600134442 shutdown_connections 2024-03-18T01:11:31.115 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:30.553+0000 7f5be55e7700 1 -- 172.21.15.29:0/2600134442 wait complete. 2024-03-18T01:11:31.115 INFO:teuthology.orchestra.run.smithi029.stdout:Generating ssh key... 2024-03-18T01:11:31.818 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:11:31 smithi029 ceph-mon[29331]: from='client.14130 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T01:11:32.741 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:11:32 smithi029 ceph-mon[29331]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T01:11:32.741 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:11:32 smithi029 ceph-mon[29331]: Generating ssh key... 2024-03-18T01:11:32.741 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:11:32 smithi029 ceph-mon[29331]: from='mgr.14116 172.21.15.29:0/340264435' entity='mgr.smithi029.qviwhl' 2024-03-18T01:11:32.741 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:11:32 smithi029 ceph-mon[29331]: from='mgr.14116 172.21.15.29:0/340264435' entity='mgr.smithi029.qviwhl' 2024-03-18T01:11:32.994 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:31.531+0000 7f74a4935700 1 Processor -- start 2024-03-18T01:11:32.994 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:31.531+0000 7f74a4935700 1 -- start start 2024-03-18T01:11:32.994 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:31.531+0000 7f74a4935700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f74a0102730 0x7f74a0102b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:11:32.994 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:31.531+0000 7f74a4935700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74a0103080 con 0x7f74a0102730 2024-03-18T01:11:32.994 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:31.531+0000 7f749e59c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f74a0102730 0x7f74a0102b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:11:32.994 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:31.532+0000 7f749e59c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f74a0102730 0x7f74a0102b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53574/0 (socket says 172.21.15.29:53574) 2024-03-18T01:11:32.994 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:31.532+0000 7f749e59c700 1 -- 172.21.15.29:0/164602891 learned_addr learned my addr 172.21.15.29:0/164602891 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-18T01:11:32.994 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:31.532+0000 7f749e59c700 1 -- 172.21.15.29:0/164602891 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74a01031c0 con 0x7f74a0102730 2024-03-18T01:11:32.994 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:31.532+0000 7f749e59c700 1 --2- 172.21.15.29:0/164602891 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f74a0102730 0x7f74a0102b40 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f7498009e50 tx=0x7f7498009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba57517b8bf28c87 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:11:32.995 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:31.532+0000 7f749d59a700 1 -- 172.21.15.29:0/164602891 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f74980092e0 con 0x7f74a0102730 2024-03-18T01:11:32.995 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:31.532+0000 7f749d59a700 1 -- 172.21.15.29:0/164602891 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f7498012430 con 0x7f74a0102730 2024-03-18T01:11:32.995 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:31.532+0000 7f749d59a700 1 -- 172.21.15.29:0/164602891 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f74980075c0 con 0x7f74a0102730 2024-03-18T01:11:32.995 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:31.533+0000 7f74a4935700 1 -- 172.21.15.29:0/164602891 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f74a0102730 msgr2=0x7f74a0102b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:11:32.995 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:31.533+0000 7f74a4935700 1 --2- 172.21.15.29:0/164602891 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f74a0102730 0x7f74a0102b40 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f7498009e50 tx=0x7f7498009840 comp rx=0 tx=0).stop 2024-03-18T01:11:32.995 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:31.533+0000 7f74a4935700 1 -- 172.21.15.29:0/164602891 shutdown_connections 2024-03-18T01:11:32.995 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:31.533+0000 7f74a4935700 1 --2- 172.21.15.29:0/164602891 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f74a0102730 0x7f74a0102b40 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:11:32.995 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:31.533+0000 7f74a4935700 1 -- 172.21.15.29:0/164602891 >> 172.21.15.29:0/164602891 conn(0x7f74a00fdf10 msgr2=0x7f74a0100320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:11:32.995 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:31.533+0000 7f74a4935700 1 -- 172.21.15.29:0/164602891 shutdown_connections 2024-03-18T01:11:32.995 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:31.533+0000 7f74a4935700 1 -- 172.21.15.29:0/164602891 wait complete. 2024-03-18T01:11:32.995 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:31.533+0000 7f74a4935700 1 Processor -- start 2024-03-18T01:11:32.995 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:31.533+0000 7f74a4935700 1 -- start start 2024-03-18T01:11:32.995 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:31.534+0000 7f74a4935700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f74a0102730 0x7f74a0111630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:11:32.995 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:31.534+0000 7f74a4935700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74a0111b70 con 0x7f74a0102730 2024-03-18T01:11:32.995 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:31.534+0000 7f749e59c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f74a0102730 0x7f74a0111630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:11:32.996 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:31.534+0000 7f749e59c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f74a0102730 0x7f74a0111630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53586/0 (socket says 172.21.15.29:53586) 2024-03-18T01:11:32.996 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:31.534+0000 7f749e59c700 1 -- 172.21.15.29:0/3960583115 learned_addr learned my addr 172.21.15.29:0/3960583115 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-18T01:11:32.996 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:31.534+0000 7f749e59c700 1 -- 172.21.15.29:0/3960583115 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7498009ae0 con 0x7f74a0102730 2024-03-18T01:11:32.996 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:31.534+0000 7f749e59c700 1 --2- 172.21.15.29:0/3960583115 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f74a0102730 0x7f74a0111630 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f7498009fd0 tx=0x7f7498009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:11:32.996 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:31.534+0000 7f748effd700 1 -- 172.21.15.29:0/3960583115 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f74980092e0 con 0x7f74a0102730 2024-03-18T01:11:32.996 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:31.534+0000 7f748effd700 1 -- 172.21.15.29:0/3960583115 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f7498007d10 con 0x7f74a0102730 2024-03-18T01:11:32.996 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:31.534+0000 7f74a4935700 1 -- 172.21.15.29:0/3960583115 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74a0111d70 con 0x7f74a0102730 2024-03-18T01:11:32.996 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:31.534+0000 7f748effd700 1 -- 172.21.15.29:0/3960583115 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f749801c8c0 con 0x7f74a0102730 2024-03-18T01:11:32.996 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:31.534+0000 7f74a4935700 1 -- 172.21.15.29:0/3960583115 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74a01128e0 con 0x7f74a0102730 2024-03-18T01:11:32.996 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:31.535+0000 7f748effd700 1 -- 172.21.15.29:0/3960583115 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7f7498019070 con 0x7f74a0102730 2024-03-18T01:11:32.996 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:31.535+0000 7f748effd700 1 --2- 172.21.15.29:0/3960583115 >> [v2:172.21.15.29:6800/950856841,v1:172.21.15.29:6801/950856841] conn(0x7f74880367e0 0x7f7488038c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:11:32.996 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:31.535+0000 7f748effd700 1 -- 172.21.15.29:0/3960583115 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f7498049d90 con 0x7f74a0102730 2024-03-18T01:11:32.996 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:31.535+0000 7f749dd9b700 1 --2- 172.21.15.29:0/3960583115 >> [v2:172.21.15.29:6800/950856841,v1:172.21.15.29:6801/950856841] conn(0x7f74880367e0 0x7f7488038c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:11:32.996 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:31.535+0000 7f74a4935700 1 -- 172.21.15.29:0/3960583115 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74a005f5f0 con 0x7f74a0102730 2024-03-18T01:11:32.996 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:31.536+0000 7f749dd9b700 1 --2- 172.21.15.29:0/3960583115 >> [v2:172.21.15.29:6800/950856841,v1:172.21.15.29:6801/950856841] conn(0x7f74880367e0 0x7f7488038c90 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f7494009990 tx=0x7f7494006e40 comp rx=0 tx=0).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:11:32.997 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:31.540+0000 7f748effd700 1 -- 172.21.15.29:0/3960583115 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175520 (secure 0 0 0) 0x7f7498017080 con 0x7f74a0102730 2024-03-18T01:11:32.997 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:31.733+0000 7f74a4935700 1 -- 172.21.15.29:0/3960583115 --> [v2:172.21.15.29:6800/950856841,v1:172.21.15.29:6801/950856841] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) v1 -- 0x7f74a00ff2e0 con 0x7f74880367e0 2024-03-18T01:11:32.997 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:32.437+0000 7f748effd700 1 -- 172.21.15.29:0/3960583115 <== mgr.14116 v2:172.21.15.29:6800/950856841 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f74a00ff2e0 con 0x7f74880367e0 2024-03-18T01:11:32.997 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:32.439+0000 7f74a4935700 1 -- 172.21.15.29:0/3960583115 >> [v2:172.21.15.29:6800/950856841,v1:172.21.15.29:6801/950856841] conn(0x7f74880367e0 msgr2=0x7f7488038c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:11:32.997 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:32.439+0000 7f74a4935700 1 --2- 172.21.15.29:0/3960583115 >> [v2:172.21.15.29:6800/950856841,v1:172.21.15.29:6801/950856841] conn(0x7f74880367e0 0x7f7488038c90 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f7494009990 tx=0x7f7494006e40 comp rx=0 tx=0).stop 2024-03-18T01:11:32.997 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:32.439+0000 7f74a4935700 1 -- 172.21.15.29:0/3960583115 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f74a0102730 msgr2=0x7f74a0111630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:11:32.997 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:32.439+0000 7f74a4935700 1 --2- 172.21.15.29:0/3960583115 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f74a0102730 0x7f74a0111630 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f7498009fd0 tx=0x7f7498009840 comp rx=0 tx=0).stop 2024-03-18T01:11:32.997 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:32.439+0000 7f74a4935700 1 -- 172.21.15.29:0/3960583115 shutdown_connections 2024-03-18T01:11:32.997 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:32.439+0000 7f74a4935700 1 --2- 172.21.15.29:0/3960583115 >> [v2:172.21.15.29:6800/950856841,v1:172.21.15.29:6801/950856841] conn(0x7f74880367e0 0x7f7488038c90 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:11:32.997 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:32.439+0000 7f74a4935700 1 --2- 172.21.15.29:0/3960583115 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f74a0102730 0x7f74a0111630 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:11:32.997 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:32.439+0000 7f74a4935700 1 -- 172.21.15.29:0/3960583115 >> 172.21.15.29:0/3960583115 conn(0x7f74a00fdf10 msgr2=0x7f74a00feb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:11:32.997 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:32.439+0000 7f74a4935700 1 -- 172.21.15.29:0/3960583115 shutdown_connections 2024-03-18T01:11:32.997 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:32.439+0000 7f74a4935700 1 -- 172.21.15.29:0/3960583115 wait complete. 2024-03-18T01:11:34.191 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:11:33 smithi029 ceph-mon[29331]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T01:11:34.200 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCi0PmGToElTL5kISz6f8jjktVyRjOxEv+aZlHENCJFo31FIHSrj08Y4ku93j5VusR0iTCsWeXc1QOE3eRqODKFOxauT/fcZqfsFobLgmyZI1soBQUjYcYFGNAyaE23RmAjil2FrsLWz+jZ1/jToiInfg6S+qyKIJ4cQecBGbYB6OfA93Hi3N3W8zDACJum2o/b6IPOk2oVRVRZnuS/lRoAI42rOZtmBWE9FXz1GG6uWgRkYQXmrZG+2eqt5AIj2NF6q6XdaOaspMPo2JISWkJRExZw4sNCbpj/xNP7zAnOYDvRaSo/rK2BE+yRSVwFM/Gd/51X2eDtkVWIitcs+DsVhAVWt8FFaIQG4YHiwDdhcv3OH5mjtEHbkQAFPiPR1EVrM66kK749iySeXXZsjxr703xejYXqMZygQHtU7XbTsXEGvDNcqCv2eZkp6WAzB1qIyGq41e87RxgSIg9zUkga6OFKZq4oLUPokZ2eIpEXRXgBVTbFqdKnGfdcPd10ZR8= ceph-37f51182-e4c4-11ee-95c9-87774f69a715 2024-03-18T01:11:34.200 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:33.404+0000 7feec8f2b700 1 Processor -- start 2024-03-18T01:11:34.200 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:33.404+0000 7feec8f2b700 1 -- start start 2024-03-18T01:11:34.200 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:33.405+0000 7feec8f2b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feec4103da0 0x7feec41041b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:11:34.200 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:33.405+0000 7feec8f2b700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feec41046f0 con 0x7feec4103da0 2024-03-18T01:11:34.200 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:33.405+0000 7feec259c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feec4103da0 0x7feec41041b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:11:34.200 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:33.405+0000 7feec259c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feec4103da0 0x7feec41041b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60182/0 (socket says 172.21.15.29:60182) 2024-03-18T01:11:34.200 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:33.405+0000 7feec259c700 1 -- 172.21.15.29:0/1361403240 learned_addr learned my addr 172.21.15.29:0/1361403240 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-18T01:11:34.200 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:33.405+0000 7feec259c700 1 -- 172.21.15.29:0/1361403240 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feec4104830 con 0x7feec4103da0 2024-03-18T01:11:34.201 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:33.405+0000 7feec259c700 1 --2- 172.21.15.29:0/1361403240 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feec4103da0 0x7feec41041b0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7feeb40098e0 tx=0x7feeb4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b67e77057783841 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:11:34.201 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:33.405+0000 7feec159a700 1 -- 172.21.15.29:0/1361403240 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7feeb400b860 con 0x7feec4103da0 2024-03-18T01:11:34.201 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:33.405+0000 7feec159a700 1 -- 172.21.15.29:0/1361403240 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7feeb400be50 con 0x7feec4103da0 2024-03-18T01:11:34.201 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:33.405+0000 7feec159a700 1 -- 172.21.15.29:0/1361403240 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7feeb4010380 con 0x7feec4103da0 2024-03-18T01:11:34.201 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:33.406+0000 7feec8f2b700 1 -- 172.21.15.29:0/1361403240 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feec4103da0 msgr2=0x7feec41041b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:11:34.201 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:33.406+0000 7feec8f2b700 1 --2- 172.21.15.29:0/1361403240 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feec4103da0 0x7feec41041b0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7feeb40098e0 tx=0x7feeb4009250 comp rx=0 tx=0).stop 2024-03-18T01:11:34.201 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:33.406+0000 7feec8f2b700 1 -- 172.21.15.29:0/1361403240 shutdown_connections 2024-03-18T01:11:34.201 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:33.406+0000 7feec8f2b700 1 --2- 172.21.15.29:0/1361403240 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feec4103da0 0x7feec41041b0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:11:34.201 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:33.406+0000 7feec8f2b700 1 -- 172.21.15.29:0/1361403240 >> 172.21.15.29:0/1361403240 conn(0x7feec40fc4a0 msgr2=0x7feec40fe8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:11:34.201 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:33.406+0000 7feec8f2b700 1 -- 172.21.15.29:0/1361403240 shutdown_connections 2024-03-18T01:11:34.201 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:33.406+0000 7feec8f2b700 1 -- 172.21.15.29:0/1361403240 wait complete. 2024-03-18T01:11:34.201 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:33.407+0000 7feec8f2b700 1 Processor -- start 2024-03-18T01:11:34.201 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:33.407+0000 7feec8f2b700 1 -- start start 2024-03-18T01:11:34.202 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:33.407+0000 7feec8f2b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feec4103da0 0x7feec4111560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:11:34.202 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:33.407+0000 7feec8f2b700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feec4111aa0 con 0x7feec4103da0 2024-03-18T01:11:34.202 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:33.407+0000 7feec259c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feec4103da0 0x7feec4111560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:11:34.202 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:33.407+0000 7feec259c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feec4103da0 0x7feec4111560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60184/0 (socket says 172.21.15.29:60184) 2024-03-18T01:11:34.202 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:33.407+0000 7feec259c700 1 -- 172.21.15.29:0/614935185 learned_addr learned my addr 172.21.15.29:0/614935185 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-18T01:11:34.202 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:33.407+0000 7feec259c700 1 -- 172.21.15.29:0/614935185 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feeb40094f0 con 0x7feec4103da0 2024-03-18T01:11:34.202 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:33.407+0000 7feec259c700 1 --2- 172.21.15.29:0/614935185 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feec4103da0 0x7feec4111560 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7feeb4000c00 tx=0x7feeb4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:11:34.202 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:33.407+0000 7feebaffd700 1 -- 172.21.15.29:0/614935185 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7feeb40107c0 con 0x7feec4103da0 2024-03-18T01:11:34.202 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:33.407+0000 7feebaffd700 1 -- 172.21.15.29:0/614935185 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7feeb4010db0 con 0x7feec4103da0 2024-03-18T01:11:34.202 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:33.407+0000 7feec8f2b700 1 -- 172.21.15.29:0/614935185 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feec4111ca0 con 0x7feec4103da0 2024-03-18T01:11:34.202 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:33.408+0000 7feebaffd700 1 -- 172.21.15.29:0/614935185 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7feeb401ad00 con 0x7feec4103da0 2024-03-18T01:11:34.202 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:33.408+0000 7feec8f2b700 1 -- 172.21.15.29:0/614935185 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feec4112810 con 0x7feec4103da0 2024-03-18T01:11:34.203 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:33.408+0000 7feebaffd700 1 -- 172.21.15.29:0/614935185 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7feeb401a530 con 0x7feec4103da0 2024-03-18T01:11:34.203 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:33.408+0000 7feec8f2b700 1 -- 172.21.15.29:0/614935185 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feec405f5f0 con 0x7feec4103da0 2024-03-18T01:11:34.203 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:33.409+0000 7feebaffd700 1 --2- 172.21.15.29:0/614935185 >> [v2:172.21.15.29:6800/950856841,v1:172.21.15.29:6801/950856841] conn(0x7feeb00367e0 0x7feeb0038c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:11:34.203 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:33.409+0000 7feebaffd700 1 -- 172.21.15.29:0/614935185 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7feeb404a650 con 0x7feec4103da0 2024-03-18T01:11:34.203 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:33.409+0000 7feec1d9b700 1 --2- 172.21.15.29:0/614935185 >> [v2:172.21.15.29:6800/950856841,v1:172.21.15.29:6801/950856841] conn(0x7feeb00367e0 0x7feeb0038c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:11:34.203 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:33.409+0000 7feec1d9b700 1 --2- 172.21.15.29:0/614935185 >> [v2:172.21.15.29:6800/950856841,v1:172.21.15.29:6801/950856841] conn(0x7feeb00367e0 0x7feeb0038c90 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7feeac009990 tx=0x7feeac006e40 comp rx=0 tx=0).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:11:34.203 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:33.413+0000 7feebaffd700 1 -- 172.21.15.29:0/614935185 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175520 (secure 0 0 0) 0x7feeb4010920 con 0x7feec4103da0 2024-03-18T01:11:34.203 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:33.606+0000 7feec8f2b700 1 -- 172.21.15.29:0/614935185 --> [v2:172.21.15.29:6800/950856841,v1:172.21.15.29:6801/950856841] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) v1 -- 0x7feec40fd750 con 0x7feeb00367e0 2024-03-18T01:11:34.203 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:33.606+0000 7feebaffd700 1 -- 172.21.15.29:0/614935185 <== mgr.14116 v2:172.21.15.29:6800/950856841 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+595 (secure 0 0 0) 0x7feec40fd750 con 0x7feeb00367e0 2024-03-18T01:11:34.203 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:33.608+0000 7feec8f2b700 1 -- 172.21.15.29:0/614935185 >> [v2:172.21.15.29:6800/950856841,v1:172.21.15.29:6801/950856841] conn(0x7feeb00367e0 msgr2=0x7feeb0038c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:11:34.203 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:33.608+0000 7feec8f2b700 1 --2- 172.21.15.29:0/614935185 >> [v2:172.21.15.29:6800/950856841,v1:172.21.15.29:6801/950856841] conn(0x7feeb00367e0 0x7feeb0038c90 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7feeac009990 tx=0x7feeac006e40 comp rx=0 tx=0).stop 2024-03-18T01:11:34.203 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:33.608+0000 7feec8f2b700 1 -- 172.21.15.29:0/614935185 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feec4103da0 msgr2=0x7feec4111560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:11:34.203 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:33.608+0000 7feec8f2b700 1 --2- 172.21.15.29:0/614935185 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feec4103da0 0x7feec4111560 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7feeb4000c00 tx=0x7feeb4009250 comp rx=0 tx=0).stop 2024-03-18T01:11:34.204 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:33.608+0000 7feec8f2b700 1 -- 172.21.15.29:0/614935185 shutdown_connections 2024-03-18T01:11:34.204 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:33.608+0000 7feec8f2b700 1 --2- 172.21.15.29:0/614935185 >> [v2:172.21.15.29:6800/950856841,v1:172.21.15.29:6801/950856841] conn(0x7feeb00367e0 0x7feeb0038c90 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:11:34.204 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:33.608+0000 7feec8f2b700 1 --2- 172.21.15.29:0/614935185 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feec4103da0 0x7feec4111560 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:11:34.204 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:33.608+0000 7feec8f2b700 1 -- 172.21.15.29:0/614935185 >> 172.21.15.29:0/614935185 conn(0x7feec40fc4a0 msgr2=0x7feec40fd040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:11:34.204 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:33.608+0000 7feec8f2b700 1 -- 172.21.15.29:0/614935185 shutdown_connections 2024-03-18T01:11:34.204 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:33.608+0000 7feec8f2b700 1 -- 172.21.15.29:0/614935185 wait complete. 2024-03-18T01:11:34.204 INFO:teuthology.orchestra.run.smithi029.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-03-18T01:11:34.204 INFO:teuthology.orchestra.run.smithi029.stdout:Adding key to root@localhost authorized_keys... 2024-03-18T01:11:34.204 INFO:teuthology.orchestra.run.smithi029.stdout:Adding host smithi029... 2024-03-18T01:11:36.318 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:11:36 smithi029 ceph-mon[29331]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi029", "addr": "172.21.15.29", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T01:11:36.319 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:11:36 smithi029 ceph-mon[29331]: Deploying cephadm binary to smithi029 2024-03-18T01:11:37.535 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout Added host 'smithi029' with addr '172.21.15.29' 2024-03-18T01:11:37.535 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:34.624+0000 7f17cefde700 1 Processor -- start 2024-03-18T01:11:37.535 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:34.624+0000 7f17cefde700 1 -- start start 2024-03-18T01:11:37.535 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:34.624+0000 7f17cefde700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f17c8078c30 0x7f17c80771e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:11:37.535 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:34.624+0000 7f17cefde700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17c8079040 con 0x7f17c8078c30 2024-03-18T01:11:37.536 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:34.624+0000 7f17ccd7a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f17c8078c30 0x7f17c80771e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:11:37.536 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:34.624+0000 7f17ccd7a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f17c8078c30 0x7f17c80771e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60190/0 (socket says 172.21.15.29:60190) 2024-03-18T01:11:37.536 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:34.624+0000 7f17ccd7a700 1 -- 172.21.15.29:0/2673336209 learned_addr learned my addr 172.21.15.29:0/2673336209 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-18T01:11:37.536 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:34.624+0000 7f17ccd7a700 1 -- 172.21.15.29:0/2673336209 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17c8077720 con 0x7f17c8078c30 2024-03-18T01:11:37.536 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:34.624+0000 7f17ccd7a700 1 --2- 172.21.15.29:0/2673336209 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f17c8078c30 0x7f17c80771e0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f17bc0098e0 tx=0x7f17bc009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=382f1b7458ab4a1b server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:11:37.536 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:34.625+0000 7f17c77fe700 1 -- 172.21.15.29:0/2673336209 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f17bc00b860 con 0x7f17c8078c30 2024-03-18T01:11:37.536 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:34.625+0000 7f17c77fe700 1 -- 172.21.15.29:0/2673336209 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f17bc00be50 con 0x7f17c8078c30 2024-03-18T01:11:37.536 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:34.625+0000 7f17c77fe700 1 -- 172.21.15.29:0/2673336209 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f17bc010380 con 0x7f17c8078c30 2024-03-18T01:11:37.536 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:34.625+0000 7f17cefde700 1 -- 172.21.15.29:0/2673336209 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f17c8078c30 msgr2=0x7f17c80771e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:11:37.536 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:34.625+0000 7f17cefde700 1 --2- 172.21.15.29:0/2673336209 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f17c8078c30 0x7f17c80771e0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f17bc0098e0 tx=0x7f17bc009250 comp rx=0 tx=0).stop 2024-03-18T01:11:37.536 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:34.625+0000 7f17cefde700 1 -- 172.21.15.29:0/2673336209 shutdown_connections 2024-03-18T01:11:37.536 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:34.625+0000 7f17cefde700 1 --2- 172.21.15.29:0/2673336209 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f17c8078c30 0x7f17c80771e0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:11:37.536 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:34.625+0000 7f17cefde700 1 -- 172.21.15.29:0/2673336209 >> 172.21.15.29:0/2673336209 conn(0x7f17c80fde90 msgr2=0x7f17c8100300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:11:37.536 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:34.625+0000 7f17cefde700 1 -- 172.21.15.29:0/2673336209 shutdown_connections 2024-03-18T01:11:37.536 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:34.625+0000 7f17cefde700 1 -- 172.21.15.29:0/2673336209 wait complete. 2024-03-18T01:11:37.537 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:34.626+0000 7f17cefde700 1 Processor -- start 2024-03-18T01:11:37.537 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:34.626+0000 7f17cefde700 1 -- start start 2024-03-18T01:11:37.537 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:34.626+0000 7f17cefde700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f17c8078c30 0x7f17c8113790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:11:37.537 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:34.626+0000 7f17cefde700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17c8113cd0 con 0x7f17c8078c30 2024-03-18T01:11:37.537 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:34.626+0000 7f17ccd7a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f17c8078c30 0x7f17c8113790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:11:37.537 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:34.626+0000 7f17ccd7a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f17c8078c30 0x7f17c8113790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60192/0 (socket says 172.21.15.29:60192) 2024-03-18T01:11:37.537 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:34.626+0000 7f17ccd7a700 1 -- 172.21.15.29:0/3960672582 learned_addr learned my addr 172.21.15.29:0/3960672582 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-18T01:11:37.537 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:34.626+0000 7f17ccd7a700 1 -- 172.21.15.29:0/3960672582 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17bc0094f0 con 0x7f17c8078c30 2024-03-18T01:11:37.537 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:34.627+0000 7f17ccd7a700 1 --2- 172.21.15.29:0/3960672582 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f17c8078c30 0x7f17c8113790 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f17bc003050 tx=0x7f17bc009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:11:37.537 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:34.627+0000 7f17c57fa700 1 -- 172.21.15.29:0/3960672582 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f17bc010760 con 0x7f17c8078c30 2024-03-18T01:11:37.537 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:34.627+0000 7f17c57fa700 1 -- 172.21.15.29:0/3960672582 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f17bc010d50 con 0x7f17c8078c30 2024-03-18T01:11:37.537 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:34.627+0000 7f17cefde700 1 -- 172.21.15.29:0/3960672582 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17c8113ed0 con 0x7f17c8078c30 2024-03-18T01:11:37.537 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:34.627+0000 7f17c57fa700 1 -- 172.21.15.29:0/3960672582 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f17bc01ae20 con 0x7f17c8078c30 2024-03-18T01:11:37.537 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:34.627+0000 7f17cefde700 1 -- 172.21.15.29:0/3960672582 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17c8114a40 con 0x7f17c8078c30 2024-03-18T01:11:37.537 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:34.628+0000 7f17c57fa700 1 -- 172.21.15.29:0/3960672582 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7f17bc02e440 con 0x7f17c8078c30 2024-03-18T01:11:37.538 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:34.628+0000 7f17cefde700 1 -- 172.21.15.29:0/3960672582 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17c805f5f0 con 0x7f17c8078c30 2024-03-18T01:11:37.538 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:34.628+0000 7f17c57fa700 1 --2- 172.21.15.29:0/3960672582 >> [v2:172.21.15.29:6800/950856841,v1:172.21.15.29:6801/950856841] conn(0x7f17b80367e0 0x7f17b8038c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:11:37.538 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:34.628+0000 7f17c57fa700 1 -- 172.21.15.29:0/3960672582 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f17bc04a750 con 0x7f17c8078c30 2024-03-18T01:11:37.538 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:34.628+0000 7f17c7fff700 1 --2- 172.21.15.29:0/3960672582 >> [v2:172.21.15.29:6800/950856841,v1:172.21.15.29:6801/950856841] conn(0x7f17b80367e0 0x7f17b8038c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:11:37.538 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:34.628+0000 7f17c7fff700 1 --2- 172.21.15.29:0/3960672582 >> [v2:172.21.15.29:6800/950856841,v1:172.21.15.29:6801/950856841] conn(0x7f17b80367e0 0x7f17b8038c90 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f17b4009990 tx=0x7f17b4006e40 comp rx=0 tx=0).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:11:37.538 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:34.633+0000 7f17c57fa700 1 -- 172.21.15.29:0/3960672582 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175520 (secure 0 0 0) 0x7f17bc018940 con 0x7f17c8078c30 2024-03-18T01:11:37.538 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:34.828+0000 7f17cefde700 1 -- 172.21.15.29:0/3960672582 --> [v2:172.21.15.29:6800/950856841,v1:172.21.15.29:6801/950856841] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi029", "addr": "172.21.15.29", "target": ["mon-mgr", ""]}) v1 -- 0x7f17c80ff220 con 0x7f17b80367e0 2024-03-18T01:11:37.538 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:36.960+0000 7f17c57fa700 1 -- 172.21.15.29:0/3960672582 <== mgr.14116 v2:172.21.15.29:6800/950856841 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+48 (secure 0 0 0) 0x7f17c80ff220 con 0x7f17b80367e0 2024-03-18T01:11:37.538 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:36.963+0000 7f17cefde700 1 -- 172.21.15.29:0/3960672582 >> [v2:172.21.15.29:6800/950856841,v1:172.21.15.29:6801/950856841] conn(0x7f17b80367e0 msgr2=0x7f17b8038c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:11:37.538 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:36.963+0000 7f17cefde700 1 --2- 172.21.15.29:0/3960672582 >> [v2:172.21.15.29:6800/950856841,v1:172.21.15.29:6801/950856841] conn(0x7f17b80367e0 0x7f17b8038c90 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f17b4009990 tx=0x7f17b4006e40 comp rx=0 tx=0).stop 2024-03-18T01:11:37.538 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:36.963+0000 7f17cefde700 1 -- 172.21.15.29:0/3960672582 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f17c8078c30 msgr2=0x7f17c8113790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:11:37.538 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:36.963+0000 7f17cefde700 1 --2- 172.21.15.29:0/3960672582 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f17c8078c30 0x7f17c8113790 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f17bc003050 tx=0x7f17bc009250 comp rx=0 tx=0).stop 2024-03-18T01:11:37.538 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:36.963+0000 7f17cefde700 1 -- 172.21.15.29:0/3960672582 shutdown_connections 2024-03-18T01:11:37.538 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:36.963+0000 7f17cefde700 1 --2- 172.21.15.29:0/3960672582 >> [v2:172.21.15.29:6800/950856841,v1:172.21.15.29:6801/950856841] conn(0x7f17b80367e0 0x7f17b8038c90 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:11:37.538 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:36.963+0000 7f17cefde700 1 --2- 172.21.15.29:0/3960672582 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f17c8078c30 0x7f17c8113790 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:11:37.539 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:36.963+0000 7f17cefde700 1 -- 172.21.15.29:0/3960672582 >> 172.21.15.29:0/3960672582 conn(0x7f17c80fde90 msgr2=0x7f17c80feab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:11:37.539 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:36.963+0000 7f17cefde700 1 -- 172.21.15.29:0/3960672582 shutdown_connections 2024-03-18T01:11:37.539 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:36.963+0000 7f17cefde700 1 -- 172.21.15.29:0/3960672582 wait complete. 2024-03-18T01:11:37.539 INFO:teuthology.orchestra.run.smithi029.stdout:Deploying mon service with default placement... 2024-03-18T01:11:38.198 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:11:37 smithi029 ceph-mon[29331]: from='mgr.14116 172.21.15.29:0/340264435' entity='mgr.smithi029.qviwhl' 2024-03-18T01:11:38.198 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:11:37 smithi029 ceph-mon[29331]: Added host smithi029 2024-03-18T01:11:38.198 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:11:37 smithi029 ceph-mon[29331]: from='mgr.14116 172.21.15.29:0/340264435' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T01:11:38.960 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-03-18T01:11:38.960 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:37.979+0000 7f836778d700 1 Processor -- start 2024-03-18T01:11:38.960 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:37.979+0000 7f836778d700 1 -- start start 2024-03-18T01:11:38.960 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:37.979+0000 7f836778d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8360076b30 0x7f8360107830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:11:38.960 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:37.979+0000 7f836778d700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8360103220 con 0x7f8360076b30 2024-03-18T01:11:38.960 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:37.979+0000 7f8365529700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8360076b30 0x7f8360107830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:11:38.960 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:37.979+0000 7f8365529700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8360076b30 0x7f8360107830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60198/0 (socket says 172.21.15.29:60198) 2024-03-18T01:11:38.961 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:37.979+0000 7f8365529700 1 -- 172.21.15.29:0/2262342084 learned_addr learned my addr 172.21.15.29:0/2262342084 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-18T01:11:38.961 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:37.979+0000 7f8365529700 1 -- 172.21.15.29:0/2262342084 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8360107d70 con 0x7f8360076b30 2024-03-18T01:11:38.961 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:37.979+0000 7f8365529700 1 --2- 172.21.15.29:0/2262342084 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8360076b30 0x7f8360107830 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f83500098e0 tx=0x7f8350009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c4e5bf2cfb40a477 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:11:38.961 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:37.980+0000 7f8357fff700 1 -- 172.21.15.29:0/2262342084 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f835000b860 con 0x7f8360076b30 2024-03-18T01:11:38.961 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:37.980+0000 7f8357fff700 1 -- 172.21.15.29:0/2262342084 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f835000be50 con 0x7f8360076b30 2024-03-18T01:11:38.961 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:37.980+0000 7f8357fff700 1 -- 172.21.15.29:0/2262342084 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8350010380 con 0x7f8360076b30 2024-03-18T01:11:38.961 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:37.980+0000 7f836778d700 1 -- 172.21.15.29:0/2262342084 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8360076b30 msgr2=0x7f8360107830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:11:38.961 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:37.980+0000 7f836778d700 1 --2- 172.21.15.29:0/2262342084 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8360076b30 0x7f8360107830 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f83500098e0 tx=0x7f8350009250 comp rx=0 tx=0).stop 2024-03-18T01:11:38.961 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:37.980+0000 7f836778d700 1 -- 172.21.15.29:0/2262342084 shutdown_connections 2024-03-18T01:11:38.961 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:37.980+0000 7f836778d700 1 --2- 172.21.15.29:0/2262342084 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8360076b30 0x7f8360107830 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:11:38.961 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:37.980+0000 7f836778d700 1 -- 172.21.15.29:0/2262342084 >> 172.21.15.29:0/2262342084 conn(0x7f83600fce40 msgr2=0x7f83600ff270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:11:38.961 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:37.980+0000 7f836778d700 1 -- 172.21.15.29:0/2262342084 shutdown_connections 2024-03-18T01:11:38.962 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:37.980+0000 7f836778d700 1 -- 172.21.15.29:0/2262342084 wait complete. 2024-03-18T01:11:38.962 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:37.981+0000 7f836778d700 1 Processor -- start 2024-03-18T01:11:38.962 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:37.981+0000 7f836778d700 1 -- start start 2024-03-18T01:11:38.962 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:37.981+0000 7f836778d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8360076b30 0x7f83601137d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:11:38.962 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:37.981+0000 7f836778d700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8360113d10 con 0x7f8360076b30 2024-03-18T01:11:38.962 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:37.981+0000 7f8365529700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8360076b30 0x7f83601137d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:11:38.962 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:37.981+0000 7f8365529700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8360076b30 0x7f83601137d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60212/0 (socket says 172.21.15.29:60212) 2024-03-18T01:11:38.962 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:37.981+0000 7f8365529700 1 -- 172.21.15.29:0/1224430661 learned_addr learned my addr 172.21.15.29:0/1224430661 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-18T01:11:38.962 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:37.981+0000 7f8365529700 1 -- 172.21.15.29:0/1224430661 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83500094f0 con 0x7f8360076b30 2024-03-18T01:11:38.962 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:37.982+0000 7f8365529700 1 --2- 172.21.15.29:0/1224430661 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8360076b30 0x7f83601137d0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f83500090b0 tx=0x7f8350009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:11:38.962 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:37.982+0000 7f8355ffb700 1 -- 172.21.15.29:0/1224430661 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8350010760 con 0x7f8360076b30 2024-03-18T01:11:38.962 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:37.982+0000 7f8355ffb700 1 -- 172.21.15.29:0/1224430661 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f8350010d50 con 0x7f8360076b30 2024-03-18T01:11:38.963 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:37.982+0000 7f836778d700 1 -- 172.21.15.29:0/1224430661 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8360113f10 con 0x7f8360076b30 2024-03-18T01:11:38.963 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:37.982+0000 7f8355ffb700 1 -- 172.21.15.29:0/1224430661 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f835001ace0 con 0x7f8360076b30 2024-03-18T01:11:38.963 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:37.982+0000 7f836778d700 1 -- 172.21.15.29:0/1224430661 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8360114a80 con 0x7f8360076b30 2024-03-18T01:11:38.963 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:37.983+0000 7f8355ffb700 1 -- 172.21.15.29:0/1224430661 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7f835002d3f0 con 0x7f8360076b30 2024-03-18T01:11:38.963 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:37.983+0000 7f8355ffb700 1 --2- 172.21.15.29:0/1224430661 >> [v2:172.21.15.29:6800/950856841,v1:172.21.15.29:6801/950856841] conn(0x7f834c036790 0x7f834c038c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:11:38.963 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:37.983+0000 7f8355ffb700 1 -- 172.21.15.29:0/1224430661 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f835004b110 con 0x7f8360076b30 2024-03-18T01:11:38.963 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:37.983+0000 7f8364d28700 1 --2- 172.21.15.29:0/1224430661 >> [v2:172.21.15.29:6800/950856841,v1:172.21.15.29:6801/950856841] conn(0x7f834c036790 0x7f834c038c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:11:38.963 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:37.983+0000 7f836778d700 1 -- 172.21.15.29:0/1224430661 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f836005f5f0 con 0x7f8360076b30 2024-03-18T01:11:38.963 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:37.983+0000 7f8364d28700 1 --2- 172.21.15.29:0/1224430661 >> [v2:172.21.15.29:6800/950856841,v1:172.21.15.29:6801/950856841] conn(0x7f834c036790 0x7f834c038c40 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f835c009990 tx=0x7f835c006e40 comp rx=0 tx=0).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:11:38.963 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:37.988+0000 7f8355ffb700 1 -- 172.21.15.29:0/1224430661 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175520 (secure 0 0 0) 0x7f835002b8e0 con 0x7f8360076b30 2024-03-18T01:11:38.963 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:38.197+0000 7f836778d700 1 -- 172.21.15.29:0/1224430661 --> [v2:172.21.15.29:6800/950856841,v1:172.21.15.29:6801/950856841] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}) v1 -- 0x7f83600fe220 con 0x7f834c036790 2024-03-18T01:11:38.963 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:38.201+0000 7f8355ffb700 1 -- 172.21.15.29:0/1224430661 <== mgr.14116 v2:172.21.15.29:6800/950856841 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f83600fe220 con 0x7f834c036790 2024-03-18T01:11:38.963 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:38.203+0000 7f836778d700 1 -- 172.21.15.29:0/1224430661 >> [v2:172.21.15.29:6800/950856841,v1:172.21.15.29:6801/950856841] conn(0x7f834c036790 msgr2=0x7f834c038c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:11:38.963 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:38.203+0000 7f836778d700 1 --2- 172.21.15.29:0/1224430661 >> [v2:172.21.15.29:6800/950856841,v1:172.21.15.29:6801/950856841] conn(0x7f834c036790 0x7f834c038c40 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f835c009990 tx=0x7f835c006e40 comp rx=0 tx=0).stop 2024-03-18T01:11:38.964 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:38.203+0000 7f836778d700 1 -- 172.21.15.29:0/1224430661 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8360076b30 msgr2=0x7f83601137d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:11:38.964 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:38.203+0000 7f836778d700 1 --2- 172.21.15.29:0/1224430661 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8360076b30 0x7f83601137d0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f83500090b0 tx=0x7f8350009250 comp rx=0 tx=0).stop 2024-03-18T01:11:38.964 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:38.203+0000 7f836778d700 1 -- 172.21.15.29:0/1224430661 shutdown_connections 2024-03-18T01:11:38.964 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:38.203+0000 7f836778d700 1 --2- 172.21.15.29:0/1224430661 >> [v2:172.21.15.29:6800/950856841,v1:172.21.15.29:6801/950856841] conn(0x7f834c036790 0x7f834c038c40 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:11:38.964 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:38.203+0000 7f836778d700 1 --2- 172.21.15.29:0/1224430661 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8360076b30 0x7f83601137d0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:11:38.964 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:38.203+0000 7f836778d700 1 -- 172.21.15.29:0/1224430661 >> 172.21.15.29:0/1224430661 conn(0x7f83600fce40 msgr2=0x7f83600fda50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:11:38.964 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:38.203+0000 7f836778d700 1 -- 172.21.15.29:0/1224430661 shutdown_connections 2024-03-18T01:11:38.964 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:38.203+0000 7f836778d700 1 -- 172.21.15.29:0/1224430661 wait complete. 2024-03-18T01:11:38.964 INFO:teuthology.orchestra.run.smithi029.stdout:Deploying mgr service with default placement... 2024-03-18T01:11:39.280 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:11:39 smithi029 ceph-mon[29331]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T01:11:39.281 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:11:39 smithi029 ceph-mon[29331]: Saving service mon spec with placement count:5 2024-03-18T01:11:39.281 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:11:39 smithi029 ceph-mon[29331]: from='mgr.14116 172.21.15.29:0/340264435' entity='mgr.smithi029.qviwhl' 2024-03-18T01:11:40.736 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:11:40 smithi029 ceph-mon[29331]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T01:11:40.736 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:11:40 smithi029 ceph-mon[29331]: Saving service mgr spec with placement count:2 2024-03-18T01:11:40.737 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:11:40 smithi029 ceph-mon[29331]: from='mgr.14116 172.21.15.29:0/340264435' entity='mgr.smithi029.qviwhl' 2024-03-18T01:11:40.737 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:11:40 smithi029 ceph-mon[29331]: from='mgr.14116 172.21.15.29:0/340264435' entity='mgr.smithi029.qviwhl' 2024-03-18T01:11:41.056 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-03-18T01:11:41.056 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:39.425+0000 7f53cd8c1700 1 Processor -- start 2024-03-18T01:11:41.056 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:39.425+0000 7f53cd8c1700 1 -- start start 2024-03-18T01:11:41.056 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:39.425+0000 7f53cd8c1700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f53c8102730 0x7f53c8102b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:11:41.056 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:39.425+0000 7f53cd8c1700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53c8103080 con 0x7f53c8102730 2024-03-18T01:11:41.056 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:39.425+0000 7f53c6ffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f53c8102730 0x7f53c8102b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:11:41.056 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:39.425+0000 7f53c6ffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f53c8102730 0x7f53c8102b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60218/0 (socket says 172.21.15.29:60218) 2024-03-18T01:11:41.056 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:39.425+0000 7f53c6ffd700 1 -- 172.21.15.29:0/2107216043 learned_addr learned my addr 172.21.15.29:0/2107216043 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-18T01:11:41.056 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:39.426+0000 7f53c6ffd700 1 -- 172.21.15.29:0/2107216043 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53c81031c0 con 0x7f53c8102730 2024-03-18T01:11:41.056 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:39.426+0000 7f53c6ffd700 1 --2- 172.21.15.29:0/2107216043 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f53c8102730 0x7f53c8102b40 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f53b80098e0 tx=0x7f53b8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84c3fbeb2a9266a8 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:11:41.056 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:39.426+0000 7f53c5ffb700 1 -- 172.21.15.29:0/2107216043 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f53b800b860 con 0x7f53c8102730 2024-03-18T01:11:41.057 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:39.426+0000 7f53c5ffb700 1 -- 172.21.15.29:0/2107216043 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f53b800be50 con 0x7f53c8102730 2024-03-18T01:11:41.057 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:39.426+0000 7f53c5ffb700 1 -- 172.21.15.29:0/2107216043 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f53b8010380 con 0x7f53c8102730 2024-03-18T01:11:41.057 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:39.427+0000 7f53cd8c1700 1 -- 172.21.15.29:0/2107216043 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f53c8102730 msgr2=0x7f53c8102b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:11:41.057 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:39.427+0000 7f53cd8c1700 1 --2- 172.21.15.29:0/2107216043 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f53c8102730 0x7f53c8102b40 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f53b80098e0 tx=0x7f53b8009250 comp rx=0 tx=0).stop 2024-03-18T01:11:41.057 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:39.427+0000 7f53cd8c1700 1 -- 172.21.15.29:0/2107216043 shutdown_connections 2024-03-18T01:11:41.057 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:39.427+0000 7f53cd8c1700 1 --2- 172.21.15.29:0/2107216043 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f53c8102730 0x7f53c8102b40 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:11:41.057 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:39.427+0000 7f53cd8c1700 1 -- 172.21.15.29:0/2107216043 >> 172.21.15.29:0/2107216043 conn(0x7f53c80fdf10 msgr2=0x7f53c8100320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:11:41.057 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:39.427+0000 7f53cd8c1700 1 -- 172.21.15.29:0/2107216043 shutdown_connections 2024-03-18T01:11:41.057 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:39.427+0000 7f53cd8c1700 1 -- 172.21.15.29:0/2107216043 wait complete. 2024-03-18T01:11:41.057 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:39.428+0000 7f53cd8c1700 1 Processor -- start 2024-03-18T01:11:41.057 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:39.428+0000 7f53cd8c1700 1 -- start start 2024-03-18T01:11:41.057 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:39.428+0000 7f53cd8c1700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f53c8102730 0x7f53c81115f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:11:41.057 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:39.428+0000 7f53cd8c1700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53c8111b30 con 0x7f53c8102730 2024-03-18T01:11:41.057 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:39.428+0000 7f53c6ffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f53c8102730 0x7f53c81115f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:11:41.057 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:39.428+0000 7f53c6ffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f53c8102730 0x7f53c81115f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60232/0 (socket says 172.21.15.29:60232) 2024-03-18T01:11:41.058 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:39.428+0000 7f53c6ffd700 1 -- 172.21.15.29:0/3852369739 learned_addr learned my addr 172.21.15.29:0/3852369739 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-18T01:11:41.058 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:39.428+0000 7f53c6ffd700 1 -- 172.21.15.29:0/3852369739 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53b80094f0 con 0x7f53c8102730 2024-03-18T01:11:41.058 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:39.428+0000 7f53c6ffd700 1 --2- 172.21.15.29:0/3852369739 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f53c8102730 0x7f53c81115f0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f53b800b440 tx=0x7f53b8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:11:41.058 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:39.428+0000 7f53af7fe700 1 -- 172.21.15.29:0/3852369739 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f53b8010760 con 0x7f53c8102730 2024-03-18T01:11:41.058 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:39.429+0000 7f53af7fe700 1 -- 172.21.15.29:0/3852369739 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f53b8010d50 con 0x7f53c8102730 2024-03-18T01:11:41.058 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:39.429+0000 7f53cd8c1700 1 -- 172.21.15.29:0/3852369739 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53c8111d30 con 0x7f53c8102730 2024-03-18T01:11:41.058 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:39.429+0000 7f53af7fe700 1 -- 172.21.15.29:0/3852369739 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f53b801adc0 con 0x7f53c8102730 2024-03-18T01:11:41.058 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:39.429+0000 7f53cd8c1700 1 -- 172.21.15.29:0/3852369739 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53c81128a0 con 0x7f53c8102730 2024-03-18T01:11:41.058 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:39.430+0000 7f53af7fe700 1 -- 172.21.15.29:0/3852369739 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7f53b8019410 con 0x7f53c8102730 2024-03-18T01:11:41.058 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:39.430+0000 7f53cd8c1700 1 -- 172.21.15.29:0/3852369739 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53c81121c0 con 0x7f53c8102730 2024-03-18T01:11:41.058 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:39.430+0000 7f53af7fe700 1 --2- 172.21.15.29:0/3852369739 >> [v2:172.21.15.29:6800/950856841,v1:172.21.15.29:6801/950856841] conn(0x7f53b0036770 0x7f53b0038c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:11:41.058 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:39.430+0000 7f53af7fe700 1 -- 172.21.15.29:0/3852369739 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f53b804a220 con 0x7f53c8102730 2024-03-18T01:11:41.058 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:39.430+0000 7f53c67fc700 1 --2- 172.21.15.29:0/3852369739 >> [v2:172.21.15.29:6800/950856841,v1:172.21.15.29:6801/950856841] conn(0x7f53b0036770 0x7f53b0038c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:11:41.058 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:39.430+0000 7f53c67fc700 1 --2- 172.21.15.29:0/3852369739 >> [v2:172.21.15.29:6800/950856841,v1:172.21.15.29:6801/950856841] conn(0x7f53b0036770 0x7f53b0038c20 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f53bc009990 tx=0x7f53bc006e40 comp rx=0 tx=0).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:11:41.059 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:39.435+0000 7f53af7fe700 1 -- 172.21.15.29:0/3852369739 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175520 (secure 0 0 0) 0x7f53b8023940 con 0x7f53c8102730 2024-03-18T01:11:41.059 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:39.643+0000 7f53cd8c1700 1 -- 172.21.15.29:0/3852369739 --> [v2:172.21.15.29:6800/950856841,v1:172.21.15.29:6801/950856841] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}) v1 -- 0x7f53c80ff2d0 con 0x7f53b0036770 2024-03-18T01:11:41.059 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:39.647+0000 7f53af7fe700 1 -- 172.21.15.29:0/3852369739 <== mgr.14116 v2:172.21.15.29:6800/950856841 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f53c80ff2d0 con 0x7f53b0036770 2024-03-18T01:11:41.059 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:39.650+0000 7f53cd8c1700 1 -- 172.21.15.29:0/3852369739 >> [v2:172.21.15.29:6800/950856841,v1:172.21.15.29:6801/950856841] conn(0x7f53b0036770 msgr2=0x7f53b0038c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:11:41.059 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:39.650+0000 7f53cd8c1700 1 --2- 172.21.15.29:0/3852369739 >> [v2:172.21.15.29:6800/950856841,v1:172.21.15.29:6801/950856841] conn(0x7f53b0036770 0x7f53b0038c20 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f53bc009990 tx=0x7f53bc006e40 comp rx=0 tx=0).stop 2024-03-18T01:11:41.059 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:39.650+0000 7f53cd8c1700 1 -- 172.21.15.29:0/3852369739 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f53c8102730 msgr2=0x7f53c81115f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:11:41.059 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:39.650+0000 7f53cd8c1700 1 --2- 172.21.15.29:0/3852369739 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f53c8102730 0x7f53c81115f0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f53b800b440 tx=0x7f53b8009250 comp rx=0 tx=0).stop 2024-03-18T01:11:41.059 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:39.650+0000 7f53cd8c1700 1 -- 172.21.15.29:0/3852369739 shutdown_connections 2024-03-18T01:11:41.059 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:39.650+0000 7f53cd8c1700 1 --2- 172.21.15.29:0/3852369739 >> [v2:172.21.15.29:6800/950856841,v1:172.21.15.29:6801/950856841] conn(0x7f53b0036770 0x7f53b0038c20 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:11:41.059 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:39.650+0000 7f53cd8c1700 1 --2- 172.21.15.29:0/3852369739 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f53c8102730 0x7f53c81115f0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:11:41.059 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:39.650+0000 7f53cd8c1700 1 -- 172.21.15.29:0/3852369739 >> 172.21.15.29:0/3852369739 conn(0x7f53c80fdf10 msgr2=0x7f53c80feb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:11:41.059 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:39.650+0000 7f53cd8c1700 1 -- 172.21.15.29:0/3852369739 shutdown_connections 2024-03-18T01:11:41.059 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:39.650+0000 7f53cd8c1700 1 -- 172.21.15.29:0/3852369739 wait complete. 2024-03-18T01:11:41.059 INFO:teuthology.orchestra.run.smithi029.stdout:Deploying crash service with default placement... 2024-03-18T01:11:42.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:11:42 smithi029 ceph-mon[29331]: from='mgr.14116 172.21.15.29:0/340264435' entity='mgr.smithi029.qviwhl' 2024-03-18T01:11:42.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:11:42 smithi029 ceph-mon[29331]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T01:11:42.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:11:42 smithi029 ceph-mon[29331]: Saving service crash spec with placement * 2024-03-18T01:11:42.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:11:42 smithi029 ceph-mon[29331]: from='mgr.14116 172.21.15.29:0/340264435' entity='mgr.smithi029.qviwhl' 2024-03-18T01:11:42.896 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout Scheduled crash update... 2024-03-18T01:11:42.896 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:41.521+0000 7fb7d3bb2700 1 Processor -- start 2024-03-18T01:11:42.896 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:41.521+0000 7fb7d3bb2700 1 -- start start 2024-03-18T01:11:42.896 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:41.522+0000 7fb7d3bb2700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb7cc1016e0 0x7fb7cc101af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:11:42.896 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:41.522+0000 7fb7d3bb2700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7cc102030 con 0x7fb7cc1016e0 2024-03-18T01:11:42.896 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:41.522+0000 7fb7d194e700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb7cc1016e0 0x7fb7cc101af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:11:42.897 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:41.522+0000 7fb7d194e700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb7cc1016e0 0x7fb7cc101af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60238/0 (socket says 172.21.15.29:60238) 2024-03-18T01:11:42.897 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:41.522+0000 7fb7d194e700 1 -- 172.21.15.29:0/517663926 learned_addr learned my addr 172.21.15.29:0/517663926 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-18T01:11:42.897 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:41.522+0000 7fb7d194e700 1 -- 172.21.15.29:0/517663926 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7cc102170 con 0x7fb7cc1016e0 2024-03-18T01:11:42.897 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:41.523+0000 7fb7d194e700 1 --2- 172.21.15.29:0/517663926 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb7cc1016e0 0x7fb7cc101af0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fb7c4009e50 tx=0x7fb7c4009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=30660592b25848ca server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:11:42.897 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:41.523+0000 7fb7d094c700 1 -- 172.21.15.29:0/517663926 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb7c40092e0 con 0x7fb7cc1016e0 2024-03-18T01:11:42.897 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:41.523+0000 7fb7d094c700 1 -- 172.21.15.29:0/517663926 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb7c4012430 con 0x7fb7cc1016e0 2024-03-18T01:11:42.897 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:41.523+0000 7fb7d094c700 1 -- 172.21.15.29:0/517663926 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb7c40075a0 con 0x7fb7cc1016e0 2024-03-18T01:11:42.897 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:41.524+0000 7fb7d3bb2700 1 -- 172.21.15.29:0/517663926 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb7cc1016e0 msgr2=0x7fb7cc101af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:11:42.897 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:41.524+0000 7fb7d3bb2700 1 --2- 172.21.15.29:0/517663926 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb7cc1016e0 0x7fb7cc101af0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fb7c4009e50 tx=0x7fb7c4009840 comp rx=0 tx=0).stop 2024-03-18T01:11:42.897 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:41.524+0000 7fb7d3bb2700 1 -- 172.21.15.29:0/517663926 shutdown_connections 2024-03-18T01:11:42.898 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:41.524+0000 7fb7d3bb2700 1 --2- 172.21.15.29:0/517663926 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb7cc1016e0 0x7fb7cc101af0 secure :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fb7c4009e50 tx=0x7fb7c4009840 comp rx=0 tx=0).stop 2024-03-18T01:11:42.898 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:41.524+0000 7fb7d3bb2700 1 -- 172.21.15.29:0/517663926 >> 172.21.15.29:0/517663926 conn(0x7fb7cc0fce40 msgr2=0x7fb7cc0ff270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:11:42.898 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:41.524+0000 7fb7d3bb2700 1 -- 172.21.15.29:0/517663926 shutdown_connections 2024-03-18T01:11:42.898 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:41.524+0000 7fb7d3bb2700 1 -- 172.21.15.29:0/517663926 wait complete. 2024-03-18T01:11:42.898 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:41.524+0000 7fb7d3bb2700 1 Processor -- start 2024-03-18T01:11:42.898 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:41.524+0000 7fb7d3bb2700 1 -- start start 2024-03-18T01:11:42.898 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:41.525+0000 7fb7d3bb2700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb7cc10f590 0x7fb7cc10f9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:11:42.898 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:41.525+0000 7fb7d3bb2700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7cc10fee0 con 0x7fb7cc10f590 2024-03-18T01:11:42.898 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:41.525+0000 7fb7d194e700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb7cc10f590 0x7fb7cc10f9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:11:42.899 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:41.525+0000 7fb7d194e700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb7cc10f590 0x7fb7cc10f9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60252/0 (socket says 172.21.15.29:60252) 2024-03-18T01:11:42.899 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:41.525+0000 7fb7d194e700 1 -- 172.21.15.29:0/3899721280 learned_addr learned my addr 172.21.15.29:0/3899721280 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-18T01:11:42.899 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:41.525+0000 7fb7d194e700 1 -- 172.21.15.29:0/3899721280 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7c4009ae0 con 0x7fb7cc10f590 2024-03-18T01:11:42.899 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:41.525+0000 7fb7d194e700 1 --2- 172.21.15.29:0/3899721280 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb7cc10f590 0x7fb7cc10f9a0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fb7c4012f20 tx=0x7fb7c4009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:11:42.899 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:41.525+0000 7fb7ca7fc700 1 -- 172.21.15.29:0/3899721280 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb7c400bd30 con 0x7fb7cc10f590 2024-03-18T01:11:42.900 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:41.526+0000 7fb7d3bb2700 1 -- 172.21.15.29:0/3899721280 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7cc1100e0 con 0x7fb7cc10f590 2024-03-18T01:11:42.900 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:41.526+0000 7fb7ca7fc700 1 -- 172.21.15.29:0/3899721280 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb7c400be90 con 0x7fb7cc10f590 2024-03-18T01:11:42.900 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:41.526+0000 7fb7ca7fc700 1 -- 172.21.15.29:0/3899721280 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb7c4026790 con 0x7fb7cc10f590 2024-03-18T01:11:42.900 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:41.526+0000 7fb7d3bb2700 1 -- 172.21.15.29:0/3899721280 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7cc110a70 con 0x7fb7cc10f590 2024-03-18T01:11:42.900 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:41.527+0000 7fb7ca7fc700 1 -- 172.21.15.29:0/3899721280 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7fb7c40268f0 con 0x7fb7cc10f590 2024-03-18T01:11:42.901 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:41.527+0000 7fb7d3bb2700 1 -- 172.21.15.29:0/3899721280 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7cc04bed0 con 0x7fb7cc10f590 2024-03-18T01:11:42.901 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:41.527+0000 7fb7ca7fc700 1 --2- 172.21.15.29:0/3899721280 >> [v2:172.21.15.29:6800/950856841,v1:172.21.15.29:6801/950856841] conn(0x7fb7b4038a30 0x7fb7b403aee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:11:42.901 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:41.527+0000 7fb7ca7fc700 1 -- 172.21.15.29:0/3899721280 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7fb7c404b2f0 con 0x7fb7cc10f590 2024-03-18T01:11:42.901 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:41.527+0000 7fb7d114d700 1 --2- 172.21.15.29:0/3899721280 >> [v2:172.21.15.29:6800/950856841,v1:172.21.15.29:6801/950856841] conn(0x7fb7b4038a30 0x7fb7b403aee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:11:42.901 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:41.528+0000 7fb7d114d700 1 --2- 172.21.15.29:0/3899721280 >> [v2:172.21.15.29:6800/950856841,v1:172.21.15.29:6801/950856841] conn(0x7fb7b4038a30 0x7fb7b403aee0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fb7c0009990 tx=0x7fb7c0006e40 comp rx=0 tx=0).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:11:42.901 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:41.532+0000 7fb7ca7fc700 1 -- 172.21.15.29:0/3899721280 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175520 (secure 0 0 0) 0x7fb7c4024d60 con 0x7fb7cc10f590 2024-03-18T01:11:42.902 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:41.768+0000 7fb7d3bb2700 1 -- 172.21.15.29:0/3899721280 --> [v2:172.21.15.29:6800/950856841,v1:172.21.15.29:6801/950856841] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}) v1 -- 0x7fb7cc1112b0 con 0x7fb7b4038a30 2024-03-18T01:11:42.902 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:41.772+0000 7fb7ca7fc700 1 -- 172.21.15.29:0/3899721280 <== mgr.14116 v2:172.21.15.29:6800/950856841 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+26 (secure 0 0 0) 0x7fb7cc1112b0 con 0x7fb7b4038a30 2024-03-18T01:11:42.902 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:41.774+0000 7fb7d3bb2700 1 -- 172.21.15.29:0/3899721280 >> [v2:172.21.15.29:6800/950856841,v1:172.21.15.29:6801/950856841] conn(0x7fb7b4038a30 msgr2=0x7fb7b403aee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:11:42.902 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:41.774+0000 7fb7d3bb2700 1 --2- 172.21.15.29:0/3899721280 >> [v2:172.21.15.29:6800/950856841,v1:172.21.15.29:6801/950856841] conn(0x7fb7b4038a30 0x7fb7b403aee0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fb7c0009990 tx=0x7fb7c0006e40 comp rx=0 tx=0).stop 2024-03-18T01:11:42.902 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:41.774+0000 7fb7d3bb2700 1 -- 172.21.15.29:0/3899721280 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb7cc10f590 msgr2=0x7fb7cc10f9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:11:42.903 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:41.774+0000 7fb7d3bb2700 1 --2- 172.21.15.29:0/3899721280 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb7cc10f590 0x7fb7cc10f9a0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fb7c4012f20 tx=0x7fb7c4009840 comp rx=0 tx=0).stop 2024-03-18T01:11:42.903 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:41.774+0000 7fb7d3bb2700 1 -- 172.21.15.29:0/3899721280 shutdown_connections 2024-03-18T01:11:42.903 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:41.774+0000 7fb7d3bb2700 1 --2- 172.21.15.29:0/3899721280 >> [v2:172.21.15.29:6800/950856841,v1:172.21.15.29:6801/950856841] conn(0x7fb7b4038a30 0x7fb7b403aee0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:11:42.903 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:41.774+0000 7fb7d3bb2700 1 --2- 172.21.15.29:0/3899721280 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb7cc10f590 0x7fb7cc10f9a0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:11:42.903 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:41.774+0000 7fb7d3bb2700 1 -- 172.21.15.29:0/3899721280 >> 172.21.15.29:0/3899721280 conn(0x7fb7cc0fce40 msgr2=0x7fb7cc105da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:11:42.904 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:41.775+0000 7fb7d3bb2700 1 -- 172.21.15.29:0/3899721280 shutdown_connections 2024-03-18T01:11:42.904 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:41.775+0000 7fb7d3bb2700 1 -- 172.21.15.29:0/3899721280 wait complete. 2024-03-18T01:11:42.904 INFO:teuthology.orchestra.run.smithi029.stdout:Deploying prometheus service with default placement... 2024-03-18T01:11:43.536 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:11:43 smithi029 ceph-mon[29331]: from='mgr.14116 172.21.15.29:0/340264435' entity='mgr.smithi029.qviwhl' 2024-03-18T01:11:43.536 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:11:43 smithi029 ceph-mon[29331]: from='mgr.14116 172.21.15.29:0/340264435' entity='mgr.smithi029.qviwhl' 2024-03-18T01:11:44.135 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout Scheduled prometheus update... 2024-03-18T01:11:44.135 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:43.302+0000 7fa4582f4700 1 Processor -- start 2024-03-18T01:11:44.135 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:43.302+0000 7fa4582f4700 1 -- start start 2024-03-18T01:11:44.135 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:43.302+0000 7fa4582f4700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa450102730 0x7fa450102b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:11:44.135 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:43.302+0000 7fa4582f4700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa450103080 con 0x7fa450102730 2024-03-18T01:11:44.136 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:43.302+0000 7fa456090700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa450102730 0x7fa450102b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:11:44.136 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:43.302+0000 7fa456090700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa450102730 0x7fa450102b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53982/0 (socket says 172.21.15.29:53982) 2024-03-18T01:11:44.136 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:43.302+0000 7fa456090700 1 -- 172.21.15.29:0/2926954070 learned_addr learned my addr 172.21.15.29:0/2926954070 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-18T01:11:44.136 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:43.302+0000 7fa456090700 1 -- 172.21.15.29:0/2926954070 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4501031c0 con 0x7fa450102730 2024-03-18T01:11:44.136 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:43.303+0000 7fa456090700 1 --2- 172.21.15.29:0/2926954070 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa450102730 0x7fa450102b40 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fa44c0098e0 tx=0x7fa44c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b1deecf8fdb6ae9e server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:11:44.136 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:43.303+0000 7fa45508e700 1 -- 172.21.15.29:0/2926954070 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa44c00b860 con 0x7fa450102730 2024-03-18T01:11:44.136 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:43.303+0000 7fa45508e700 1 -- 172.21.15.29:0/2926954070 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa44c00be90 con 0x7fa450102730 2024-03-18T01:11:44.136 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:43.303+0000 7fa45508e700 1 -- 172.21.15.29:0/2926954070 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa44c0103c0 con 0x7fa450102730 2024-03-18T01:11:44.136 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:43.303+0000 7fa4582f4700 1 -- 172.21.15.29:0/2926954070 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa450102730 msgr2=0x7fa450102b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:11:44.136 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:43.303+0000 7fa4582f4700 1 --2- 172.21.15.29:0/2926954070 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa450102730 0x7fa450102b40 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fa44c0098e0 tx=0x7fa44c009250 comp rx=0 tx=0).stop 2024-03-18T01:11:44.136 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:43.303+0000 7fa4582f4700 1 -- 172.21.15.29:0/2926954070 shutdown_connections 2024-03-18T01:11:44.136 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:43.303+0000 7fa4582f4700 1 --2- 172.21.15.29:0/2926954070 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa450102730 0x7fa450102b40 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:11:44.136 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:43.303+0000 7fa4582f4700 1 -- 172.21.15.29:0/2926954070 >> 172.21.15.29:0/2926954070 conn(0x7fa4500fdf10 msgr2=0x7fa450100320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:11:44.136 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:43.304+0000 7fa4582f4700 1 -- 172.21.15.29:0/2926954070 shutdown_connections 2024-03-18T01:11:44.136 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:43.304+0000 7fa4582f4700 1 -- 172.21.15.29:0/2926954070 wait complete. 2024-03-18T01:11:44.136 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:43.304+0000 7fa4582f4700 1 Processor -- start 2024-03-18T01:11:44.137 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:43.304+0000 7fa4582f4700 1 -- start start 2024-03-18T01:11:44.137 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:43.304+0000 7fa4582f4700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa450102730 0x7fa450111630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:11:44.137 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:43.304+0000 7fa4582f4700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa450111b70 con 0x7fa450102730 2024-03-18T01:11:44.137 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:43.304+0000 7fa456090700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa450102730 0x7fa450111630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:11:44.137 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:43.304+0000 7fa456090700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa450102730 0x7fa450111630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53990/0 (socket says 172.21.15.29:53990) 2024-03-18T01:11:44.137 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:43.304+0000 7fa456090700 1 -- 172.21.15.29:0/3915241905 learned_addr learned my addr 172.21.15.29:0/3915241905 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-18T01:11:44.137 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:43.305+0000 7fa456090700 1 -- 172.21.15.29:0/3915241905 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa44c0094f0 con 0x7fa450102730 2024-03-18T01:11:44.137 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:43.305+0000 7fa456090700 1 --2- 172.21.15.29:0/3915241905 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa450102730 0x7fa450111630 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fa44c0094c0 tx=0x7fa44c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:11:44.137 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:43.305+0000 7fa446ffd700 1 -- 172.21.15.29:0/3915241905 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa44c010680 con 0x7fa450102730 2024-03-18T01:11:44.137 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:43.305+0000 7fa446ffd700 1 -- 172.21.15.29:0/3915241905 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa44c010cb0 con 0x7fa450102730 2024-03-18T01:11:44.137 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:43.305+0000 7fa4582f4700 1 -- 172.21.15.29:0/3915241905 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa450111d70 con 0x7fa450102730 2024-03-18T01:11:44.137 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:43.305+0000 7fa446ffd700 1 -- 172.21.15.29:0/3915241905 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa44c01ad40 con 0x7fa450102730 2024-03-18T01:11:44.137 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:43.305+0000 7fa4582f4700 1 -- 172.21.15.29:0/3915241905 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4501128e0 con 0x7fa450102730 2024-03-18T01:11:44.137 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:43.306+0000 7fa446ffd700 1 -- 172.21.15.29:0/3915241905 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7fa44c0107e0 con 0x7fa450102730 2024-03-18T01:11:44.137 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:43.306+0000 7fa446ffd700 1 --2- 172.21.15.29:0/3915241905 >> [v2:172.21.15.29:6800/950856841,v1:172.21.15.29:6801/950856841] conn(0x7fa43c0367e0 0x7fa43c038c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:11:44.137 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:43.306+0000 7fa446ffd700 1 -- 172.21.15.29:0/3915241905 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7fa44c04a270 con 0x7fa450102730 2024-03-18T01:11:44.138 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:43.306+0000 7fa4582f4700 1 -- 172.21.15.29:0/3915241905 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa45005f5f0 con 0x7fa450102730 2024-03-18T01:11:44.138 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:43.306+0000 7fa45588f700 1 --2- 172.21.15.29:0/3915241905 >> [v2:172.21.15.29:6800/950856841,v1:172.21.15.29:6801/950856841] conn(0x7fa43c0367e0 0x7fa43c038c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:11:44.138 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:43.307+0000 7fa45588f700 1 --2- 172.21.15.29:0/3915241905 >> [v2:172.21.15.29:6800/950856841,v1:172.21.15.29:6801/950856841] conn(0x7fa43c0367e0 0x7fa43c038c90 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fa440009990 tx=0x7fa440006e40 comp rx=0 tx=0).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:11:44.138 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:43.311+0000 7fa446ffd700 1 -- 172.21.15.29:0/3915241905 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175520 (secure 0 0 0) 0x7fa44c020070 con 0x7fa450102730 2024-03-18T01:11:44.138 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:43.519+0000 7fa4582f4700 1 -- 172.21.15.29:0/3915241905 --> [v2:172.21.15.29:6800/950856841,v1:172.21.15.29:6801/950856841] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}) v1 -- 0x7fa4500ff2d0 con 0x7fa43c0367e0 2024-03-18T01:11:44.138 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:43.523+0000 7fa446ffd700 1 -- 172.21.15.29:0/3915241905 <== mgr.14116 v2:172.21.15.29:6800/950856841 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+31 (secure 0 0 0) 0x7fa4500ff2d0 con 0x7fa43c0367e0 2024-03-18T01:11:44.138 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:43.525+0000 7fa4582f4700 1 -- 172.21.15.29:0/3915241905 >> [v2:172.21.15.29:6800/950856841,v1:172.21.15.29:6801/950856841] conn(0x7fa43c0367e0 msgr2=0x7fa43c038c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:11:44.138 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:43.525+0000 7fa4582f4700 1 --2- 172.21.15.29:0/3915241905 >> [v2:172.21.15.29:6800/950856841,v1:172.21.15.29:6801/950856841] conn(0x7fa43c0367e0 0x7fa43c038c90 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fa440009990 tx=0x7fa440006e40 comp rx=0 tx=0).stop 2024-03-18T01:11:44.138 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:43.525+0000 7fa4582f4700 1 -- 172.21.15.29:0/3915241905 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa450102730 msgr2=0x7fa450111630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:11:44.138 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:43.525+0000 7fa4582f4700 1 --2- 172.21.15.29:0/3915241905 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa450102730 0x7fa450111630 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fa44c0094c0 tx=0x7fa44c009250 comp rx=0 tx=0).stop 2024-03-18T01:11:44.138 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:43.525+0000 7fa4582f4700 1 -- 172.21.15.29:0/3915241905 shutdown_connections 2024-03-18T01:11:44.138 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:43.525+0000 7fa4582f4700 1 --2- 172.21.15.29:0/3915241905 >> [v2:172.21.15.29:6800/950856841,v1:172.21.15.29:6801/950856841] conn(0x7fa43c0367e0 0x7fa43c038c90 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:11:44.138 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:43.525+0000 7fa4582f4700 1 --2- 172.21.15.29:0/3915241905 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa450102730 0x7fa450111630 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:11:44.138 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:43.525+0000 7fa4582f4700 1 -- 172.21.15.29:0/3915241905 >> 172.21.15.29:0/3915241905 conn(0x7fa4500fdf10 msgr2=0x7fa4500feb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:11:44.138 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:43.525+0000 7fa4582f4700 1 -- 172.21.15.29:0/3915241905 shutdown_connections 2024-03-18T01:11:44.139 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:43.525+0000 7fa4582f4700 1 -- 172.21.15.29:0/3915241905 wait complete. 2024-03-18T01:11:44.139 INFO:teuthology.orchestra.run.smithi029.stdout:Deploying grafana service with default placement... 2024-03-18T01:11:44.796 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:11:44 smithi029 ceph-mon[29331]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T01:11:44.797 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:11:44 smithi029 ceph-mon[29331]: Saving service prometheus spec with placement count:1 2024-03-18T01:11:44.797 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:11:44 smithi029 ceph-mon[29331]: from='mgr.14116 172.21.15.29:0/340264435' entity='mgr.smithi029.qviwhl' 2024-03-18T01:11:45.878 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:11:45 smithi029 ceph-mon[29331]: from='client.14146 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T01:11:45.878 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:11:45 smithi029 ceph-mon[29331]: Saving service grafana spec with placement count:1 2024-03-18T01:11:45.878 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:11:45 smithi029 ceph-mon[29331]: from='mgr.14116 172.21.15.29:0/340264435' entity='mgr.smithi029.qviwhl' 2024-03-18T01:11:45.885 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout Scheduled grafana update... 2024-03-18T01:11:45.885 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:44.618+0000 7f715e0aa700 1 Processor -- start 2024-03-18T01:11:45.885 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:44.618+0000 7f715e0aa700 1 -- start start 2024-03-18T01:11:45.885 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:44.618+0000 7f715e0aa700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7158102be0 0x7f7158102ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:11:45.886 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:44.618+0000 7f715e0aa700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7158103530 con 0x7f7158102be0 2024-03-18T01:11:45.886 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:44.619+0000 7f71577fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7158102be0 0x7f7158102ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:11:45.886 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:44.619+0000 7f71577fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7158102be0 0x7f7158102ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54000/0 (socket says 172.21.15.29:54000) 2024-03-18T01:11:45.886 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:44.619+0000 7f71577fe700 1 -- 172.21.15.29:0/4127702788 learned_addr learned my addr 172.21.15.29:0/4127702788 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-18T01:11:45.886 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:44.619+0000 7f71577fe700 1 -- 172.21.15.29:0/4127702788 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7158103670 con 0x7f7158102be0 2024-03-18T01:11:45.886 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:44.619+0000 7f71577fe700 1 --2- 172.21.15.29:0/4127702788 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7158102be0 0x7f7158102ff0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f7150009e50 tx=0x7f7150009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75cffb45fbb53553 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:11:45.886 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:44.619+0000 7f71567fc700 1 -- 172.21.15.29:0/4127702788 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f71500092e0 con 0x7f7158102be0 2024-03-18T01:11:45.886 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:44.619+0000 7f71567fc700 1 -- 172.21.15.29:0/4127702788 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7150012430 con 0x7f7158102be0 2024-03-18T01:11:45.886 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:44.619+0000 7f71567fc700 1 -- 172.21.15.29:0/4127702788 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f71500075a0 con 0x7f7158102be0 2024-03-18T01:11:45.886 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:44.620+0000 7f715e0aa700 1 -- 172.21.15.29:0/4127702788 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7158102be0 msgr2=0x7f7158102ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:11:45.886 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:44.620+0000 7f715e0aa700 1 --2- 172.21.15.29:0/4127702788 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7158102be0 0x7f7158102ff0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f7150009e50 tx=0x7f7150009840 comp rx=0 tx=0).stop 2024-03-18T01:11:45.886 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:44.620+0000 7f715e0aa700 1 -- 172.21.15.29:0/4127702788 shutdown_connections 2024-03-18T01:11:45.886 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:44.620+0000 7f715e0aa700 1 --2- 172.21.15.29:0/4127702788 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7158102be0 0x7f7158102ff0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:11:45.886 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:44.620+0000 7f715e0aa700 1 -- 172.21.15.29:0/4127702788 >> 172.21.15.29:0/4127702788 conn(0x7f71580fc500 msgr2=0x7f71580fe910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:11:45.886 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:44.620+0000 7f715e0aa700 1 -- 172.21.15.29:0/4127702788 shutdown_connections 2024-03-18T01:11:45.887 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:44.620+0000 7f715e0aa700 1 -- 172.21.15.29:0/4127702788 wait complete. 2024-03-18T01:11:45.887 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:44.621+0000 7f715e0aa700 1 Processor -- start 2024-03-18T01:11:45.887 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:44.621+0000 7f715e0aa700 1 -- start start 2024-03-18T01:11:45.887 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:44.621+0000 7f715e0aa700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7158102be0 0x7f71581013b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:11:45.887 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:44.621+0000 7f715e0aa700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71581142c0 con 0x7f7158102be0 2024-03-18T01:11:45.887 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:44.621+0000 7f71577fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7158102be0 0x7f71581013b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:11:45.887 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:44.621+0000 7f71577fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7158102be0 0x7f71581013b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54010/0 (socket says 172.21.15.29:54010) 2024-03-18T01:11:45.887 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:44.621+0000 7f71577fe700 1 -- 172.21.15.29:0/4214204256 learned_addr learned my addr 172.21.15.29:0/4214204256 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-18T01:11:45.887 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:44.621+0000 7f71577fe700 1 -- 172.21.15.29:0/4214204256 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7150009ae0 con 0x7f7158102be0 2024-03-18T01:11:45.887 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:44.621+0000 7f71577fe700 1 --2- 172.21.15.29:0/4214204256 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7158102be0 0x7f71581013b0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f71500127d0 tx=0x7f7150009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:11:45.887 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:44.622+0000 7f714ffff700 1 -- 172.21.15.29:0/4214204256 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f71500092e0 con 0x7f7158102be0 2024-03-18T01:11:45.887 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:44.622+0000 7f714ffff700 1 -- 172.21.15.29:0/4214204256 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7150007d30 con 0x7f7158102be0 2024-03-18T01:11:45.887 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:44.622+0000 7f715e0aa700 1 -- 172.21.15.29:0/4214204256 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7158114400 con 0x7f7158102be0 2024-03-18T01:11:45.887 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:44.622+0000 7f715e0aa700 1 -- 172.21.15.29:0/4214204256 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7158114890 con 0x7f7158102be0 2024-03-18T01:11:45.887 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:44.622+0000 7f714ffff700 1 -- 172.21.15.29:0/4214204256 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7150025960 con 0x7f7158102be0 2024-03-18T01:11:45.887 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:44.623+0000 7f715e0aa700 1 -- 172.21.15.29:0/4214204256 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f715804bed0 con 0x7f7158102be0 2024-03-18T01:11:45.888 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:44.623+0000 7f714ffff700 1 -- 172.21.15.29:0/4214204256 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7f7150019050 con 0x7f7158102be0 2024-03-18T01:11:45.888 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:44.623+0000 7f714ffff700 1 --2- 172.21.15.29:0/4214204256 >> [v2:172.21.15.29:6800/950856841,v1:172.21.15.29:6801/950856841] conn(0x7f71440367e0 0x7f7144038c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:11:45.888 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:44.623+0000 7f714ffff700 1 -- 172.21.15.29:0/4214204256 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f715004a1f0 con 0x7f7158102be0 2024-03-18T01:11:45.888 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:44.624+0000 7f7156ffd700 1 --2- 172.21.15.29:0/4214204256 >> [v2:172.21.15.29:6800/950856841,v1:172.21.15.29:6801/950856841] conn(0x7f71440367e0 0x7f7144038c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:11:45.888 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:44.624+0000 7f7156ffd700 1 --2- 172.21.15.29:0/4214204256 >> [v2:172.21.15.29:6800/950856841,v1:172.21.15.29:6801/950856841] conn(0x7f71440367e0 0x7f7144038c90 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f7140009990 tx=0x7f7140006e40 comp rx=0 tx=0).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:11:45.888 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:44.628+0000 7f714ffff700 1 -- 172.21.15.29:0/4214204256 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175520 (secure 0 0 0) 0x7f7150011570 con 0x7f7158102be0 2024-03-18T01:11:45.888 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:44.858+0000 7f715e0aa700 1 -- 172.21.15.29:0/4214204256 --> [v2:172.21.15.29:6800/950856841,v1:172.21.15.29:6801/950856841] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}) v1 -- 0x7f7158114ad0 con 0x7f71440367e0 2024-03-18T01:11:45.888 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:44.862+0000 7f714ffff700 1 -- 172.21.15.29:0/4214204256 <== mgr.14116 v2:172.21.15.29:6800/950856841 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+28 (secure 0 0 0) 0x7f7158114ad0 con 0x7f71440367e0 2024-03-18T01:11:45.888 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:44.864+0000 7f715e0aa700 1 -- 172.21.15.29:0/4214204256 >> [v2:172.21.15.29:6800/950856841,v1:172.21.15.29:6801/950856841] conn(0x7f71440367e0 msgr2=0x7f7144038c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:11:45.888 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:44.864+0000 7f715e0aa700 1 --2- 172.21.15.29:0/4214204256 >> [v2:172.21.15.29:6800/950856841,v1:172.21.15.29:6801/950856841] conn(0x7f71440367e0 0x7f7144038c90 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f7140009990 tx=0x7f7140006e40 comp rx=0 tx=0).stop 2024-03-18T01:11:45.888 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:44.864+0000 7f715e0aa700 1 -- 172.21.15.29:0/4214204256 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7158102be0 msgr2=0x7f71581013b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:11:45.888 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:44.864+0000 7f715e0aa700 1 --2- 172.21.15.29:0/4214204256 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7158102be0 0x7f71581013b0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f71500127d0 tx=0x7f7150009840 comp rx=0 tx=0).stop 2024-03-18T01:11:45.888 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:44.864+0000 7f715e0aa700 1 -- 172.21.15.29:0/4214204256 shutdown_connections 2024-03-18T01:11:45.888 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:44.864+0000 7f715e0aa700 1 --2- 172.21.15.29:0/4214204256 >> [v2:172.21.15.29:6800/950856841,v1:172.21.15.29:6801/950856841] conn(0x7f71440367e0 0x7f7144038c90 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:11:45.888 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:44.864+0000 7f715e0aa700 1 --2- 172.21.15.29:0/4214204256 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7158102be0 0x7f71581013b0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:11:45.889 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:44.864+0000 7f715e0aa700 1 -- 172.21.15.29:0/4214204256 >> 172.21.15.29:0/4214204256 conn(0x7f71580fc500 msgr2=0x7f71580fd0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:11:45.889 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:44.864+0000 7f715e0aa700 1 -- 172.21.15.29:0/4214204256 shutdown_connections 2024-03-18T01:11:45.889 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:44.864+0000 7f715e0aa700 1 -- 172.21.15.29:0/4214204256 wait complete. 2024-03-18T01:11:45.889 INFO:teuthology.orchestra.run.smithi029.stdout:Deploying node-exporter service with default placement... 2024-03-18T01:11:47.407 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout Scheduled node-exporter update... 2024-03-18T01:11:47.407 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:46.333+0000 7ff21e171700 1 Processor -- start 2024-03-18T01:11:47.407 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:46.333+0000 7ff21e171700 1 -- start start 2024-03-18T01:11:47.407 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:46.333+0000 7ff21e171700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff218076c50 0x7ff2181052a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:11:47.407 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:46.333+0000 7ff21e171700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2181057e0 con 0x7ff218076c50 2024-03-18T01:11:47.407 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:46.333+0000 7ff2177fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff218076c50 0x7ff2181052a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:11:47.407 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:46.333+0000 7ff2177fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff218076c50 0x7ff2181052a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54020/0 (socket says 172.21.15.29:54020) 2024-03-18T01:11:47.407 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:46.333+0000 7ff2177fe700 1 -- 172.21.15.29:0/1215546594 learned_addr learned my addr 172.21.15.29:0/1215546594 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-18T01:11:47.407 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:46.334+0000 7ff2177fe700 1 -- 172.21.15.29:0/1215546594 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff218105920 con 0x7ff218076c50 2024-03-18T01:11:47.407 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:46.334+0000 7ff2177fe700 1 --2- 172.21.15.29:0/1215546594 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff218076c50 0x7ff2181052a0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7ff2080098e0 tx=0x7ff208009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=198880a33b49359b server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:11:47.408 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:46.334+0000 7ff2167fc700 1 -- 172.21.15.29:0/1215546594 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff20800b860 con 0x7ff218076c50 2024-03-18T01:11:47.408 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:46.334+0000 7ff2167fc700 1 -- 172.21.15.29:0/1215546594 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff20800be90 con 0x7ff218076c50 2024-03-18T01:11:47.408 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:46.334+0000 7ff2167fc700 1 -- 172.21.15.29:0/1215546594 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff2080103c0 con 0x7ff218076c50 2024-03-18T01:11:47.408 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:46.335+0000 7ff21e171700 1 -- 172.21.15.29:0/1215546594 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff218076c50 msgr2=0x7ff2181052a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:11:47.408 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:46.335+0000 7ff21e171700 1 --2- 172.21.15.29:0/1215546594 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff218076c50 0x7ff2181052a0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7ff2080098e0 tx=0x7ff208009250 comp rx=0 tx=0).stop 2024-03-18T01:11:47.408 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:46.335+0000 7ff21e171700 1 -- 172.21.15.29:0/1215546594 shutdown_connections 2024-03-18T01:11:47.408 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:46.335+0000 7ff21e171700 1 --2- 172.21.15.29:0/1215546594 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff218076c50 0x7ff2181052a0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:11:47.408 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:46.335+0000 7ff21e171700 1 -- 172.21.15.29:0/1215546594 >> 172.21.15.29:0/1215546594 conn(0x7ff2180fcd50 msgr2=0x7ff2180ff160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:11:47.408 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:46.335+0000 7ff21e171700 1 -- 172.21.15.29:0/1215546594 shutdown_connections 2024-03-18T01:11:47.408 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:46.335+0000 7ff21e171700 1 -- 172.21.15.29:0/1215546594 wait complete. 2024-03-18T01:11:47.408 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:46.335+0000 7ff21e171700 1 Processor -- start 2024-03-18T01:11:47.408 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:46.336+0000 7ff21e171700 1 -- start start 2024-03-18T01:11:47.408 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:46.336+0000 7ff21e171700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff218076c50 0x7ff2181115b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:11:47.408 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:46.336+0000 7ff21e171700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff218111af0 con 0x7ff218076c50 2024-03-18T01:11:47.408 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:46.336+0000 7ff2177fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff218076c50 0x7ff2181115b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:11:47.409 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:46.336+0000 7ff2177fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff218076c50 0x7ff2181115b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54024/0 (socket says 172.21.15.29:54024) 2024-03-18T01:11:47.409 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:46.336+0000 7ff2177fe700 1 -- 172.21.15.29:0/2628457143 learned_addr learned my addr 172.21.15.29:0/2628457143 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-18T01:11:47.409 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:46.336+0000 7ff2177fe700 1 -- 172.21.15.29:0/2628457143 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2080094f0 con 0x7ff218076c50 2024-03-18T01:11:47.409 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:46.337+0000 7ff2177fe700 1 --2- 172.21.15.29:0/2628457143 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff218076c50 0x7ff2181115b0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7ff2080094c0 tx=0x7ff208009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:11:47.409 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:46.337+0000 7ff20ffff700 1 -- 172.21.15.29:0/2628457143 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff208010890 con 0x7ff218076c50 2024-03-18T01:11:47.409 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:46.337+0000 7ff21e171700 1 -- 172.21.15.29:0/2628457143 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff218111cf0 con 0x7ff218076c50 2024-03-18T01:11:47.409 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:46.337+0000 7ff20ffff700 1 -- 172.21.15.29:0/2628457143 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff208010ec0 con 0x7ff218076c50 2024-03-18T01:11:47.409 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:46.337+0000 7ff20ffff700 1 -- 172.21.15.29:0/2628457143 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff2080243f0 con 0x7ff218076c50 2024-03-18T01:11:47.409 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:46.337+0000 7ff21e171700 1 -- 172.21.15.29:0/2628457143 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff218112860 con 0x7ff218076c50 2024-03-18T01:11:47.409 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:46.338+0000 7ff20ffff700 1 -- 172.21.15.29:0/2628457143 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7ff20802d3f0 con 0x7ff218076c50 2024-03-18T01:11:47.409 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:46.338+0000 7ff20ffff700 1 --2- 172.21.15.29:0/2628457143 >> [v2:172.21.15.29:6800/950856841,v1:172.21.15.29:6801/950856841] conn(0x7ff204036790 0x7ff204038c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:11:47.409 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:46.338+0000 7ff21e171700 1 -- 172.21.15.29:0/2628457143 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff21805f5f0 con 0x7ff218076c50 2024-03-18T01:11:47.409 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:46.338+0000 7ff20ffff700 1 -- 172.21.15.29:0/2628457143 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7ff20804a2d0 con 0x7ff218076c50 2024-03-18T01:11:47.409 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:46.338+0000 7ff216ffd700 1 --2- 172.21.15.29:0/2628457143 >> [v2:172.21.15.29:6800/950856841,v1:172.21.15.29:6801/950856841] conn(0x7ff204036790 0x7ff204038c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:11:47.410 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:46.339+0000 7ff216ffd700 1 --2- 172.21.15.29:0/2628457143 >> [v2:172.21.15.29:6800/950856841,v1:172.21.15.29:6801/950856841] conn(0x7ff204036790 0x7ff204038c40 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7ff200009990 tx=0x7ff200006e40 comp rx=0 tx=0).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:11:47.410 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:46.343+0000 7ff20ffff700 1 -- 172.21.15.29:0/2628457143 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175520 (secure 0 0 0) 0x7ff20802b8e0 con 0x7ff218076c50 2024-03-18T01:11:47.410 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:46.548+0000 7ff21e171700 1 -- 172.21.15.29:0/2628457143 --> [v2:172.21.15.29:6800/950856841,v1:172.21.15.29:6801/950856841] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}) v1 -- 0x7ff2180fe040 con 0x7ff204036790 2024-03-18T01:11:47.410 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:46.553+0000 7ff20ffff700 1 -- 172.21.15.29:0/2628457143 <== mgr.14116 v2:172.21.15.29:6800/950856841 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+34 (secure 0 0 0) 0x7ff2180fe040 con 0x7ff204036790 2024-03-18T01:11:47.410 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:46.555+0000 7ff21e171700 1 -- 172.21.15.29:0/2628457143 >> [v2:172.21.15.29:6800/950856841,v1:172.21.15.29:6801/950856841] conn(0x7ff204036790 msgr2=0x7ff204038c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:11:47.410 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:46.555+0000 7ff21e171700 1 --2- 172.21.15.29:0/2628457143 >> [v2:172.21.15.29:6800/950856841,v1:172.21.15.29:6801/950856841] conn(0x7ff204036790 0x7ff204038c40 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7ff200009990 tx=0x7ff200006e40 comp rx=0 tx=0).stop 2024-03-18T01:11:47.410 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:46.555+0000 7ff21e171700 1 -- 172.21.15.29:0/2628457143 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff218076c50 msgr2=0x7ff2181115b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:11:47.410 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:46.555+0000 7ff21e171700 1 --2- 172.21.15.29:0/2628457143 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff218076c50 0x7ff2181115b0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7ff2080094c0 tx=0x7ff208009250 comp rx=0 tx=0).stop 2024-03-18T01:11:47.410 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:46.555+0000 7ff21e171700 1 -- 172.21.15.29:0/2628457143 shutdown_connections 2024-03-18T01:11:47.410 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:46.555+0000 7ff21e171700 1 --2- 172.21.15.29:0/2628457143 >> [v2:172.21.15.29:6800/950856841,v1:172.21.15.29:6801/950856841] conn(0x7ff204036790 0x7ff204038c40 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:11:47.410 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:46.555+0000 7ff21e171700 1 --2- 172.21.15.29:0/2628457143 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff218076c50 0x7ff2181115b0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:11:47.410 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:46.555+0000 7ff21e171700 1 -- 172.21.15.29:0/2628457143 >> 172.21.15.29:0/2628457143 conn(0x7ff2180fcd50 msgr2=0x7ff2180fd8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:11:47.410 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:46.555+0000 7ff21e171700 1 -- 172.21.15.29:0/2628457143 shutdown_connections 2024-03-18T01:11:47.410 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:46.555+0000 7ff21e171700 1 -- 172.21.15.29:0/2628457143 wait complete. 2024-03-18T01:11:47.411 INFO:teuthology.orchestra.run.smithi029.stdout:Deploying alertmanager service with default placement... 2024-03-18T01:11:47.684 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:11:47 smithi029 ceph-mon[29331]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T01:11:47.684 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:11:47 smithi029 ceph-mon[29331]: from='client.14148 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T01:11:47.684 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:11:47 smithi029 ceph-mon[29331]: Saving service node-exporter spec with placement * 2024-03-18T01:11:47.684 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:11:47 smithi029 ceph-mon[29331]: from='mgr.14116 172.21.15.29:0/340264435' entity='mgr.smithi029.qviwhl' 2024-03-18T01:11:48.682 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout Scheduled alertmanager update... 2024-03-18T01:11:48.682 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:47.886+0000 7f2c132fb700 1 Processor -- start 2024-03-18T01:11:48.682 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:47.886+0000 7f2c132fb700 1 -- start start 2024-03-18T01:11:48.682 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:47.886+0000 7f2c132fb700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2c0c102d10 0x7f2c0c103120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:11:48.683 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:47.886+0000 7f2c132fb700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c0c103660 con 0x7f2c0c102d10 2024-03-18T01:11:48.683 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:47.887+0000 7f2c11097700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2c0c102d10 0x7f2c0c103120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:11:48.683 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:47.887+0000 7f2c11097700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2c0c102d10 0x7f2c0c103120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54038/0 (socket says 172.21.15.29:54038) 2024-03-18T01:11:48.683 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:47.887+0000 7f2c11097700 1 -- 172.21.15.29:0/2975667368 learned_addr learned my addr 172.21.15.29:0/2975667368 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-18T01:11:48.683 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:47.887+0000 7f2c11097700 1 -- 172.21.15.29:0/2975667368 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c0c1037a0 con 0x7f2c0c102d10 2024-03-18T01:11:48.683 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:47.887+0000 7f2c11097700 1 --2- 172.21.15.29:0/2975667368 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2c0c102d10 0x7f2c0c103120 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f2c04009e50 tx=0x7f2c04009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59435e0e469bfdda server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:11:48.683 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:47.887+0000 7f2c0bfff700 1 -- 172.21.15.29:0/2975667368 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2c040092e0 con 0x7f2c0c102d10 2024-03-18T01:11:48.683 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:47.887+0000 7f2c0bfff700 1 -- 172.21.15.29:0/2975667368 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2c04012430 con 0x7f2c0c102d10 2024-03-18T01:11:48.683 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:47.887+0000 7f2c0bfff700 1 -- 172.21.15.29:0/2975667368 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2c040075a0 con 0x7f2c0c102d10 2024-03-18T01:11:48.683 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:47.888+0000 7f2c132fb700 1 -- 172.21.15.29:0/2975667368 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2c0c102d10 msgr2=0x7f2c0c103120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:11:48.683 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:47.888+0000 7f2c132fb700 1 --2- 172.21.15.29:0/2975667368 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2c0c102d10 0x7f2c0c103120 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f2c04009e50 tx=0x7f2c04009840 comp rx=0 tx=0).stop 2024-03-18T01:11:48.683 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:47.888+0000 7f2c132fb700 1 -- 172.21.15.29:0/2975667368 shutdown_connections 2024-03-18T01:11:48.683 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:47.888+0000 7f2c132fb700 1 --2- 172.21.15.29:0/2975667368 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2c0c102d10 0x7f2c0c103120 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:11:48.684 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:47.888+0000 7f2c132fb700 1 -- 172.21.15.29:0/2975667368 >> 172.21.15.29:0/2975667368 conn(0x7f2c0c0fc500 msgr2=0x7f2c0c0fe910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:11:48.684 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:47.888+0000 7f2c132fb700 1 -- 172.21.15.29:0/2975667368 shutdown_connections 2024-03-18T01:11:48.684 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:47.888+0000 7f2c132fb700 1 -- 172.21.15.29:0/2975667368 wait complete. 2024-03-18T01:11:48.684 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:47.889+0000 7f2c132fb700 1 Processor -- start 2024-03-18T01:11:48.684 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:47.889+0000 7f2c132fb700 1 -- start start 2024-03-18T01:11:48.684 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:47.889+0000 7f2c132fb700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2c0c102d10 0x7f2c0c1137e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:11:48.684 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:47.889+0000 7f2c132fb700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c0c113d20 con 0x7f2c0c102d10 2024-03-18T01:11:48.684 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:47.889+0000 7f2c11097700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2c0c102d10 0x7f2c0c1137e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:11:48.684 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:47.889+0000 7f2c11097700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2c0c102d10 0x7f2c0c1137e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54048/0 (socket says 172.21.15.29:54048) 2024-03-18T01:11:48.684 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:47.889+0000 7f2c11097700 1 -- 172.21.15.29:0/2214545191 learned_addr learned my addr 172.21.15.29:0/2214545191 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-18T01:11:48.684 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:47.889+0000 7f2c11097700 1 -- 172.21.15.29:0/2214545191 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c04009ae0 con 0x7f2c0c102d10 2024-03-18T01:11:48.684 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:47.889+0000 7f2c11097700 1 --2- 172.21.15.29:0/2214545191 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2c0c102d10 0x7f2c0c1137e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f2c04009fd0 tx=0x7f2c04009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:11:48.685 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:47.890+0000 7f2c09ffb700 1 -- 172.21.15.29:0/2214545191 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2c040092e0 con 0x7f2c0c102d10 2024-03-18T01:11:48.685 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:47.890+0000 7f2c09ffb700 1 -- 172.21.15.29:0/2214545191 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2c04007ce0 con 0x7f2c0c102d10 2024-03-18T01:11:48.685 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:47.890+0000 7f2c09ffb700 1 -- 172.21.15.29:0/2214545191 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2c04025990 con 0x7f2c0c102d10 2024-03-18T01:11:48.685 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:47.890+0000 7f2c132fb700 1 -- 172.21.15.29:0/2214545191 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c0c113f20 con 0x7f2c0c102d10 2024-03-18T01:11:48.685 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:47.890+0000 7f2c132fb700 1 -- 172.21.15.29:0/2214545191 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c0c114a90 con 0x7f2c0c102d10 2024-03-18T01:11:48.685 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:47.891+0000 7f2c09ffb700 1 -- 172.21.15.29:0/2214545191 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7f2c04019050 con 0x7f2c0c102d10 2024-03-18T01:11:48.686 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:47.891+0000 7f2c132fb700 1 -- 172.21.15.29:0/2214545191 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c0c05f5f0 con 0x7f2c0c102d10 2024-03-18T01:11:48.686 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:47.891+0000 7f2c09ffb700 1 --2- 172.21.15.29:0/2214545191 >> [v2:172.21.15.29:6800/950856841,v1:172.21.15.29:6801/950856841] conn(0x7f2bf40367e0 0x7f2bf4038c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:11:48.686 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:47.891+0000 7f2c09ffb700 1 -- 172.21.15.29:0/2214545191 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f2c0404a050 con 0x7f2c0c102d10 2024-03-18T01:11:48.686 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:47.891+0000 7f2c10896700 1 --2- 172.21.15.29:0/2214545191 >> [v2:172.21.15.29:6800/950856841,v1:172.21.15.29:6801/950856841] conn(0x7f2bf40367e0 0x7f2bf4038c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:11:48.686 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:47.892+0000 7f2c10896700 1 --2- 172.21.15.29:0/2214545191 >> [v2:172.21.15.29:6800/950856841,v1:172.21.15.29:6801/950856841] conn(0x7f2bf40367e0 0x7f2bf4038c90 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f2c00009990 tx=0x7f2c00006e40 comp rx=0 tx=0).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:11:48.687 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:47.896+0000 7f2c09ffb700 1 -- 172.21.15.29:0/2214545191 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175520 (secure 0 0 0) 0x7f2c0400be30 con 0x7f2c0c102d10 2024-03-18T01:11:48.687 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:48.097+0000 7f2c132fb700 1 -- 172.21.15.29:0/2214545191 --> [v2:172.21.15.29:6800/950856841,v1:172.21.15.29:6801/950856841] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}) v1 -- 0x7f2c0c0fd7e0 con 0x7f2bf40367e0 2024-03-18T01:11:48.687 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:48.101+0000 7f2c09ffb700 1 -- 172.21.15.29:0/2214545191 <== mgr.14116 v2:172.21.15.29:6800/950856841 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+33 (secure 0 0 0) 0x7f2c0c0fd7e0 con 0x7f2bf40367e0 2024-03-18T01:11:48.687 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:48.103+0000 7f2c132fb700 1 -- 172.21.15.29:0/2214545191 >> [v2:172.21.15.29:6800/950856841,v1:172.21.15.29:6801/950856841] conn(0x7f2bf40367e0 msgr2=0x7f2bf4038c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:11:48.687 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:48.103+0000 7f2c132fb700 1 --2- 172.21.15.29:0/2214545191 >> [v2:172.21.15.29:6800/950856841,v1:172.21.15.29:6801/950856841] conn(0x7f2bf40367e0 0x7f2bf4038c90 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f2c00009990 tx=0x7f2c00006e40 comp rx=0 tx=0).stop 2024-03-18T01:11:48.687 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:48.103+0000 7f2c132fb700 1 -- 172.21.15.29:0/2214545191 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2c0c102d10 msgr2=0x7f2c0c1137e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:11:48.688 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:48.103+0000 7f2c132fb700 1 --2- 172.21.15.29:0/2214545191 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2c0c102d10 0x7f2c0c1137e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f2c04009fd0 tx=0x7f2c04009840 comp rx=0 tx=0).stop 2024-03-18T01:11:48.688 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:48.103+0000 7f2c132fb700 1 -- 172.21.15.29:0/2214545191 shutdown_connections 2024-03-18T01:11:48.688 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:48.103+0000 7f2c132fb700 1 --2- 172.21.15.29:0/2214545191 >> [v2:172.21.15.29:6800/950856841,v1:172.21.15.29:6801/950856841] conn(0x7f2bf40367e0 0x7f2bf4038c90 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:11:48.688 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:48.103+0000 7f2c132fb700 1 --2- 172.21.15.29:0/2214545191 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2c0c102d10 0x7f2c0c1137e0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:11:48.688 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:48.103+0000 7f2c132fb700 1 -- 172.21.15.29:0/2214545191 >> 172.21.15.29:0/2214545191 conn(0x7f2c0c0fc500 msgr2=0x7f2c0c0fd070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:11:48.688 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:48.103+0000 7f2c132fb700 1 -- 172.21.15.29:0/2214545191 shutdown_connections 2024-03-18T01:11:48.688 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:48.103+0000 7f2c132fb700 1 -- 172.21.15.29:0/2214545191 wait complete. 2024-03-18T01:11:49.280 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:11:49 smithi029 ceph-mon[29331]: from='client.14150 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T01:11:49.281 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:11:49 smithi029 ceph-mon[29331]: Saving service alertmanager spec with placement count:1 2024-03-18T01:11:49.281 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:11:49 smithi029 ceph-mon[29331]: from='mgr.14116 172.21.15.29:0/340264435' entity='mgr.smithi029.qviwhl' 2024-03-18T01:11:49.281 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:11:49 smithi029 ceph-mon[29331]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T01:11:49.818 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:49.079+0000 7f193110d700 1 Processor -- start 2024-03-18T01:11:49.818 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:49.080+0000 7f193110d700 1 -- start start 2024-03-18T01:11:49.818 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:49.080+0000 7f193110d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f192c104a70 0x7f192c104e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:11:49.819 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:49.080+0000 7f193110d700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f192c105450 con 0x7f192c104a70 2024-03-18T01:11:49.819 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:49.080+0000 7f192ad9d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f192c104a70 0x7f192c104e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:11:49.819 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:49.080+0000 7f192ad9d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f192c104a70 0x7f192c104e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54064/0 (socket says 172.21.15.29:54064) 2024-03-18T01:11:49.819 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:49.080+0000 7f192ad9d700 1 -- 172.21.15.29:0/1766318941 learned_addr learned my addr 172.21.15.29:0/1766318941 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-18T01:11:49.819 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:49.080+0000 7f192ad9d700 1 -- 172.21.15.29:0/1766318941 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f192c105590 con 0x7f192c104a70 2024-03-18T01:11:49.819 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:49.080+0000 7f192ad9d700 1 --2- 172.21.15.29:0/1766318941 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f192c104a70 0x7f192c104e80 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f191c0098e0 tx=0x7f191c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d4498926fe867b4 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:11:49.819 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:49.080+0000 7f1929d9b700 1 -- 172.21.15.29:0/1766318941 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f191c00b860 con 0x7f192c104a70 2024-03-18T01:11:49.819 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:49.080+0000 7f1929d9b700 1 -- 172.21.15.29:0/1766318941 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f191c00be90 con 0x7f192c104a70 2024-03-18T01:11:49.819 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:49.080+0000 7f1929d9b700 1 -- 172.21.15.29:0/1766318941 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f191c0103c0 con 0x7f192c104a70 2024-03-18T01:11:49.819 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:49.081+0000 7f193110d700 1 -- 172.21.15.29:0/1766318941 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f192c104a70 msgr2=0x7f192c104e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:11:49.819 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:49.081+0000 7f193110d700 1 --2- 172.21.15.29:0/1766318941 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f192c104a70 0x7f192c104e80 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f191c0098e0 tx=0x7f191c009250 comp rx=0 tx=0).stop 2024-03-18T01:11:49.819 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:49.081+0000 7f193110d700 1 -- 172.21.15.29:0/1766318941 shutdown_connections 2024-03-18T01:11:49.819 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:49.081+0000 7f193110d700 1 --2- 172.21.15.29:0/1766318941 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f192c104a70 0x7f192c104e80 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:11:49.819 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:49.081+0000 7f193110d700 1 -- 172.21.15.29:0/1766318941 >> 172.21.15.29:0/1766318941 conn(0x7f192c078280 msgr2=0x7f192c0786c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:11:49.819 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:49.081+0000 7f193110d700 1 -- 172.21.15.29:0/1766318941 shutdown_connections 2024-03-18T01:11:49.820 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:49.081+0000 7f193110d700 1 -- 172.21.15.29:0/1766318941 wait complete. 2024-03-18T01:11:49.820 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:49.082+0000 7f193110d700 1 Processor -- start 2024-03-18T01:11:49.820 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:49.082+0000 7f193110d700 1 -- start start 2024-03-18T01:11:49.820 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:49.082+0000 7f193110d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f192c104a70 0x7f192c113750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:11:49.820 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:49.082+0000 7f193110d700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f192c113c90 con 0x7f192c104a70 2024-03-18T01:11:49.820 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:49.082+0000 7f192ad9d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f192c104a70 0x7f192c113750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:11:49.820 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:49.082+0000 7f192ad9d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f192c104a70 0x7f192c113750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54076/0 (socket says 172.21.15.29:54076) 2024-03-18T01:11:49.820 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:49.082+0000 7f192ad9d700 1 -- 172.21.15.29:0/4155424619 learned_addr learned my addr 172.21.15.29:0/4155424619 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-18T01:11:49.820 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:49.082+0000 7f192ad9d700 1 -- 172.21.15.29:0/4155424619 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f191c0094f0 con 0x7f192c104a70 2024-03-18T01:11:49.820 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:49.082+0000 7f192ad9d700 1 --2- 172.21.15.29:0/4155424619 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f192c104a70 0x7f192c113750 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f191c0090b0 tx=0x7f191c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:11:49.820 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:49.082+0000 7f19137fe700 1 -- 172.21.15.29:0/4155424619 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f191c010840 con 0x7f192c104a70 2024-03-18T01:11:49.820 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:49.083+0000 7f19137fe700 1 -- 172.21.15.29:0/4155424619 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f191c010e70 con 0x7f192c104a70 2024-03-18T01:11:49.820 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:49.083+0000 7f19137fe700 1 -- 172.21.15.29:0/4155424619 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f191c024e40 con 0x7f192c104a70 2024-03-18T01:11:49.820 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:49.083+0000 7f193110d700 1 -- 172.21.15.29:0/4155424619 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f192c113e90 con 0x7f192c104a70 2024-03-18T01:11:49.820 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:49.083+0000 7f193110d700 1 -- 172.21.15.29:0/4155424619 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f192c114a00 con 0x7f192c104a70 2024-03-18T01:11:49.821 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:49.083+0000 7f19137fe700 1 -- 172.21.15.29:0/4155424619 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7f191c0109a0 con 0x7f192c104a70 2024-03-18T01:11:49.821 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:49.084+0000 7f193110d700 1 -- 172.21.15.29:0/4155424619 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f192c05f5f0 con 0x7f192c104a70 2024-03-18T01:11:49.821 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:49.084+0000 7f19137fe700 1 --2- 172.21.15.29:0/4155424619 >> [v2:172.21.15.29:6800/950856841,v1:172.21.15.29:6801/950856841] conn(0x7f1914036790 0x7f1914038c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:11:49.821 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:49.084+0000 7f19137fe700 1 -- 172.21.15.29:0/4155424619 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f191c04a7c0 con 0x7f192c104a70 2024-03-18T01:11:49.821 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:49.084+0000 7f192a59c700 1 --2- 172.21.15.29:0/4155424619 >> [v2:172.21.15.29:6800/950856841,v1:172.21.15.29:6801/950856841] conn(0x7f1914036790 0x7f1914038c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:11:49.821 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:49.084+0000 7f192a59c700 1 --2- 172.21.15.29:0/4155424619 >> [v2:172.21.15.29:6800/950856841,v1:172.21.15.29:6801/950856841] conn(0x7f1914036790 0x7f1914038c40 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f1920009990 tx=0x7f1920006e40 comp rx=0 tx=0).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:11:49.821 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:49.089+0000 7f19137fe700 1 -- 172.21.15.29:0/4155424619 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175520 (secure 0 0 0) 0x7f191c02c520 con 0x7f192c104a70 2024-03-18T01:11:49.821 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:49.280+0000 7f193110d700 1 -- 172.21.15.29:0/4155424619 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) v1 -- 0x7f192c04bed0 con 0x7f192c104a70 2024-03-18T01:11:49.821 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:49.283+0000 7f19137fe700 1 -- 172.21.15.29:0/4155424619 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v7)=0 v7) v1 ==== 142+0+0 (secure 0 0 0) 0x7f191c02c340 con 0x7f192c104a70 2024-03-18T01:11:49.821 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:49.285+0000 7f193110d700 1 -- 172.21.15.29:0/4155424619 >> [v2:172.21.15.29:6800/950856841,v1:172.21.15.29:6801/950856841] conn(0x7f1914036790 msgr2=0x7f1914038c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:11:49.821 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:49.285+0000 7f193110d700 1 --2- 172.21.15.29:0/4155424619 >> [v2:172.21.15.29:6800/950856841,v1:172.21.15.29:6801/950856841] conn(0x7f1914036790 0x7f1914038c40 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f1920009990 tx=0x7f1920006e40 comp rx=0 tx=0).stop 2024-03-18T01:11:49.821 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:49.285+0000 7f193110d700 1 -- 172.21.15.29:0/4155424619 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f192c104a70 msgr2=0x7f192c113750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:11:49.821 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:49.285+0000 7f193110d700 1 --2- 172.21.15.29:0/4155424619 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f192c104a70 0x7f192c113750 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f191c0090b0 tx=0x7f191c009250 comp rx=0 tx=0).stop 2024-03-18T01:11:49.821 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:49.285+0000 7f193110d700 1 -- 172.21.15.29:0/4155424619 shutdown_connections 2024-03-18T01:11:49.821 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:49.285+0000 7f193110d700 1 --2- 172.21.15.29:0/4155424619 >> [v2:172.21.15.29:6800/950856841,v1:172.21.15.29:6801/950856841] conn(0x7f1914036790 0x7f1914038c40 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:11:49.821 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:49.285+0000 7f193110d700 1 --2- 172.21.15.29:0/4155424619 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f192c104a70 0x7f192c113750 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:11:49.822 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:49.285+0000 7f193110d700 1 -- 172.21.15.29:0/4155424619 >> 172.21.15.29:0/4155424619 conn(0x7f192c078280 msgr2=0x7f192c102440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:11:49.822 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:49.285+0000 7f193110d700 1 -- 172.21.15.29:0/4155424619 shutdown_connections 2024-03-18T01:11:49.822 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:49.285+0000 7f193110d700 1 -- 172.21.15.29:0/4155424619 wait complete. 2024-03-18T01:11:50.430 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:11:50 smithi029 ceph-mon[29331]: from='client.? 172.21.15.29:0/4155424619' entity='client.admin' 2024-03-18T01:11:51.034 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:50.229+0000 7f52a5714700 1 Processor -- start 2024-03-18T01:11:51.035 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:50.229+0000 7f52a5714700 1 -- start start 2024-03-18T01:11:51.035 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:50.229+0000 7f52a5714700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f52a0102bb0 0x7f52a0102fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:11:51.035 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:50.229+0000 7f52a5714700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52a0103500 con 0x7f52a0102bb0 2024-03-18T01:11:51.035 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:50.230+0000 7f529effd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f52a0102bb0 0x7f52a0102fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:11:51.035 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:50.230+0000 7f529effd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f52a0102bb0 0x7f52a0102fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54092/0 (socket says 172.21.15.29:54092) 2024-03-18T01:11:51.035 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:50.230+0000 7f529effd700 1 -- 172.21.15.29:0/625517275 learned_addr learned my addr 172.21.15.29:0/625517275 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-18T01:11:51.035 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:50.230+0000 7f529effd700 1 -- 172.21.15.29:0/625517275 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52a0103640 con 0x7f52a0102bb0 2024-03-18T01:11:51.035 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:50.230+0000 7f529effd700 1 --2- 172.21.15.29:0/625517275 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f52a0102bb0 0x7f52a0102fc0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f52980098e0 tx=0x7f5298009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea91d09e6beb2c10 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:11:51.035 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:50.230+0000 7f529dffb700 1 -- 172.21.15.29:0/625517275 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f529800b860 con 0x7f52a0102bb0 2024-03-18T01:11:51.035 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:50.230+0000 7f529dffb700 1 -- 172.21.15.29:0/625517275 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f529800be90 con 0x7f52a0102bb0 2024-03-18T01:11:51.035 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:50.230+0000 7f529dffb700 1 -- 172.21.15.29:0/625517275 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f52980103c0 con 0x7f52a0102bb0 2024-03-18T01:11:51.035 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:50.231+0000 7f52a5714700 1 -- 172.21.15.29:0/625517275 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f52a0102bb0 msgr2=0x7f52a0102fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:11:51.035 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:50.231+0000 7f52a5714700 1 --2- 172.21.15.29:0/625517275 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f52a0102bb0 0x7f52a0102fc0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f52980098e0 tx=0x7f5298009250 comp rx=0 tx=0).stop 2024-03-18T01:11:51.035 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:50.231+0000 7f52a5714700 1 -- 172.21.15.29:0/625517275 shutdown_connections 2024-03-18T01:11:51.036 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:50.231+0000 7f52a5714700 1 --2- 172.21.15.29:0/625517275 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f52a0102bb0 0x7f52a0102fc0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:11:51.036 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:50.231+0000 7f52a5714700 1 -- 172.21.15.29:0/625517275 >> 172.21.15.29:0/625517275 conn(0x7f52a00fc4d0 msgr2=0x7f52a00fe8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:11:51.036 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:50.231+0000 7f52a5714700 1 -- 172.21.15.29:0/625517275 shutdown_connections 2024-03-18T01:11:51.036 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:50.231+0000 7f52a5714700 1 -- 172.21.15.29:0/625517275 wait complete. 2024-03-18T01:11:51.036 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:50.231+0000 7f52a5714700 1 Processor -- start 2024-03-18T01:11:51.036 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:50.232+0000 7f52a5714700 1 -- start start 2024-03-18T01:11:51.036 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:50.232+0000 7f52a5714700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f52a0102bb0 0x7f52a0101380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:11:51.036 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:50.232+0000 7f52a5714700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52a01142d0 con 0x7f52a0102bb0 2024-03-18T01:11:51.036 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:50.232+0000 7f529effd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f52a0102bb0 0x7f52a0101380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:11:51.036 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:50.232+0000 7f529effd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f52a0102bb0 0x7f52a0101380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54102/0 (socket says 172.21.15.29:54102) 2024-03-18T01:11:51.036 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:50.232+0000 7f529effd700 1 -- 172.21.15.29:0/2919136553 learned_addr learned my addr 172.21.15.29:0/2919136553 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-18T01:11:51.036 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:50.232+0000 7f529effd700 1 -- 172.21.15.29:0/2919136553 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52980094f0 con 0x7f52a0102bb0 2024-03-18T01:11:51.036 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:50.232+0000 7f529effd700 1 --2- 172.21.15.29:0/2919136553 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f52a0102bb0 0x7f52a0101380 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f5298009fd0 tx=0x7f5298009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:11:51.036 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:50.232+0000 7f52977fe700 1 -- 172.21.15.29:0/2919136553 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5298010680 con 0x7f52a0102bb0 2024-03-18T01:11:51.036 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:50.232+0000 7f52977fe700 1 -- 172.21.15.29:0/2919136553 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5298010cb0 con 0x7f52a0102bb0 2024-03-18T01:11:51.037 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:50.232+0000 7f52a5714700 1 -- 172.21.15.29:0/2919136553 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52a0114410 con 0x7f52a0102bb0 2024-03-18T01:11:51.037 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:50.232+0000 7f52977fe700 1 -- 172.21.15.29:0/2919136553 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f529801acf0 con 0x7f52a0102bb0 2024-03-18T01:11:51.037 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:50.232+0000 7f52a5714700 1 -- 172.21.15.29:0/2919136553 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52a01148a0 con 0x7f52a0102bb0 2024-03-18T01:11:51.037 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:50.233+0000 7f52977fe700 1 -- 172.21.15.29:0/2919136553 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7f52980193c0 con 0x7f52a0102bb0 2024-03-18T01:11:51.037 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:50.234+0000 7f52a5714700 1 -- 172.21.15.29:0/2919136553 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52a004bed0 con 0x7f52a0102bb0 2024-03-18T01:11:51.037 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:50.234+0000 7f52977fe700 1 --2- 172.21.15.29:0/2919136553 >> [v2:172.21.15.29:6800/950856841,v1:172.21.15.29:6801/950856841] conn(0x7f528c036790 0x7f528c038c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:11:51.037 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:50.234+0000 7f52977fe700 1 -- 172.21.15.29:0/2919136553 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f5298049fc0 con 0x7f52a0102bb0 2024-03-18T01:11:51.037 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:50.234+0000 7f529e7fc700 1 --2- 172.21.15.29:0/2919136553 >> [v2:172.21.15.29:6800/950856841,v1:172.21.15.29:6801/950856841] conn(0x7f528c036790 0x7f528c038c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:11:51.037 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:50.234+0000 7f529e7fc700 1 --2- 172.21.15.29:0/2919136553 >> [v2:172.21.15.29:6800/950856841,v1:172.21.15.29:6801/950856841] conn(0x7f528c036790 0x7f528c038c40 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f5288009990 tx=0x7f5288006e40 comp rx=0 tx=0).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:11:51.037 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:50.238+0000 7f52977fe700 1 -- 172.21.15.29:0/2919136553 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175520 (secure 0 0 0) 0x7f5298023710 con 0x7f52a0102bb0 2024-03-18T01:11:51.037 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:50.430+0000 7f52a5714700 1 -- 172.21.15.29:0/2919136553 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) v1 -- 0x7f52a0102440 con 0x7f52a0102bb0 2024-03-18T01:11:51.037 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:50.433+0000 7f52977fe700 1 -- 172.21.15.29:0/2919136553 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v8)=0 v8) v1 ==== 130+0+0 (secure 0 0 0) 0x7f5298020200 con 0x7f52a0102bb0 2024-03-18T01:11:51.037 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:50.435+0000 7f52a5714700 1 -- 172.21.15.29:0/2919136553 >> [v2:172.21.15.29:6800/950856841,v1:172.21.15.29:6801/950856841] conn(0x7f528c036790 msgr2=0x7f528c038c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:11:51.037 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:50.435+0000 7f52a5714700 1 --2- 172.21.15.29:0/2919136553 >> [v2:172.21.15.29:6800/950856841,v1:172.21.15.29:6801/950856841] conn(0x7f528c036790 0x7f528c038c40 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f5288009990 tx=0x7f5288006e40 comp rx=0 tx=0).stop 2024-03-18T01:11:51.037 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:50.435+0000 7f52a5714700 1 -- 172.21.15.29:0/2919136553 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f52a0102bb0 msgr2=0x7f52a0101380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:11:51.038 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:50.435+0000 7f52a5714700 1 --2- 172.21.15.29:0/2919136553 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f52a0102bb0 0x7f52a0101380 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f5298009fd0 tx=0x7f5298009250 comp rx=0 tx=0).stop 2024-03-18T01:11:51.038 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:50.435+0000 7f52a5714700 1 -- 172.21.15.29:0/2919136553 shutdown_connections 2024-03-18T01:11:51.038 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:50.435+0000 7f52a5714700 1 --2- 172.21.15.29:0/2919136553 >> [v2:172.21.15.29:6800/950856841,v1:172.21.15.29:6801/950856841] conn(0x7f528c036790 0x7f528c038c40 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:11:51.038 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:50.435+0000 7f52a5714700 1 --2- 172.21.15.29:0/2919136553 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f52a0102bb0 0x7f52a0101380 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:11:51.038 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:50.435+0000 7f52a5714700 1 -- 172.21.15.29:0/2919136553 >> 172.21.15.29:0/2919136553 conn(0x7f52a00fc4d0 msgr2=0x7f52a00fd010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:11:51.038 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:50.435+0000 7f52a5714700 1 -- 172.21.15.29:0/2919136553 shutdown_connections 2024-03-18T01:11:51.038 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:50.435+0000 7f52a5714700 1 -- 172.21.15.29:0/2919136553 wait complete. 2024-03-18T01:11:51.038 INFO:teuthology.orchestra.run.smithi029.stdout:Enabling the dashboard module... 2024-03-18T01:11:51.749 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:11:51 smithi029 ceph-mon[29331]: from='client.? 172.21.15.29:0/2919136553' entity='client.admin' 2024-03-18T01:11:51.749 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:11:51 smithi029 ceph-mon[29331]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T01:11:52.756 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:11:52 smithi029 ceph-mon[29331]: from='client.? 172.21.15.29:0/1681601058' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-03-18T01:11:53.069 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:51.494+0000 7f565cfd1700 1 Processor -- start 2024-03-18T01:11:53.069 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:51.495+0000 7f565cfd1700 1 -- start start 2024-03-18T01:11:53.069 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:51.495+0000 7f565cfd1700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5658076be0 0x7f5658105310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:11:53.069 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:51.495+0000 7f565cfd1700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5658105850 con 0x7f5658076be0 2024-03-18T01:11:53.069 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:51.495+0000 7f565659c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5658076be0 0x7f5658105310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:11:53.069 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:51.495+0000 7f565659c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5658076be0 0x7f5658105310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54104/0 (socket says 172.21.15.29:54104) 2024-03-18T01:11:53.069 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:51.495+0000 7f565659c700 1 -- 172.21.15.29:0/4199053206 learned_addr learned my addr 172.21.15.29:0/4199053206 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-18T01:11:53.069 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:51.495+0000 7f565659c700 1 -- 172.21.15.29:0/4199053206 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5658105990 con 0x7f5658076be0 2024-03-18T01:11:53.069 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:51.496+0000 7f565659c700 1 --2- 172.21.15.29:0/4199053206 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5658076be0 0x7f5658105310 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f56480098e0 tx=0x7f5648009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b1290ac8b6647a5e server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:11:53.069 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:51.496+0000 7f565559a700 1 -- 172.21.15.29:0/4199053206 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f564800b860 con 0x7f5658076be0 2024-03-18T01:11:53.069 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:51.496+0000 7f565559a700 1 -- 172.21.15.29:0/4199053206 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f564800be90 con 0x7f5658076be0 2024-03-18T01:11:53.070 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:51.496+0000 7f565559a700 1 -- 172.21.15.29:0/4199053206 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f56480103c0 con 0x7f5658076be0 2024-03-18T01:11:53.070 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:51.496+0000 7f565cfd1700 1 -- 172.21.15.29:0/4199053206 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5658076be0 msgr2=0x7f5658105310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:11:53.070 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:51.496+0000 7f565cfd1700 1 --2- 172.21.15.29:0/4199053206 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5658076be0 0x7f5658105310 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f56480098e0 tx=0x7f5648009250 comp rx=0 tx=0).stop 2024-03-18T01:11:53.070 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:51.496+0000 7f565cfd1700 1 -- 172.21.15.29:0/4199053206 shutdown_connections 2024-03-18T01:11:53.070 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:51.496+0000 7f565cfd1700 1 --2- 172.21.15.29:0/4199053206 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5658076be0 0x7f5658105310 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:11:53.070 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:51.496+0000 7f565cfd1700 1 -- 172.21.15.29:0/4199053206 >> 172.21.15.29:0/4199053206 conn(0x7f56580fcdc0 msgr2=0x7f56580ff1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:11:53.070 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:51.496+0000 7f565cfd1700 1 -- 172.21.15.29:0/4199053206 shutdown_connections 2024-03-18T01:11:53.070 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:51.496+0000 7f565cfd1700 1 -- 172.21.15.29:0/4199053206 wait complete. 2024-03-18T01:11:53.070 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:51.497+0000 7f565cfd1700 1 Processor -- start 2024-03-18T01:11:53.070 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:51.497+0000 7f565cfd1700 1 -- start start 2024-03-18T01:11:53.070 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:51.497+0000 7f565cfd1700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5658076be0 0x7f56581115b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:11:53.070 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:51.497+0000 7f565cfd1700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5658111af0 con 0x7f5658076be0 2024-03-18T01:11:53.070 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:51.497+0000 7f565659c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5658076be0 0x7f56581115b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:11:53.070 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:51.497+0000 7f565659c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5658076be0 0x7f56581115b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54108/0 (socket says 172.21.15.29:54108) 2024-03-18T01:11:53.071 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:51.497+0000 7f565659c700 1 -- 172.21.15.29:0/1681601058 learned_addr learned my addr 172.21.15.29:0/1681601058 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-18T01:11:53.071 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:51.497+0000 7f565659c700 1 -- 172.21.15.29:0/1681601058 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56480094f0 con 0x7f5658076be0 2024-03-18T01:11:53.071 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:51.498+0000 7f565659c700 1 --2- 172.21.15.29:0/1681601058 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5658076be0 0x7f56581115b0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f5648006d40 tx=0x7f5648009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:11:53.071 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:51.498+0000 7f5646ffd700 1 -- 172.21.15.29:0/1681601058 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5648010840 con 0x7f5658076be0 2024-03-18T01:11:53.071 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:51.498+0000 7f5646ffd700 1 -- 172.21.15.29:0/1681601058 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5648010e70 con 0x7f5658076be0 2024-03-18T01:11:53.071 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:51.498+0000 7f565cfd1700 1 -- 172.21.15.29:0/1681601058 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5658111cf0 con 0x7f5658076be0 2024-03-18T01:11:53.071 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:51.498+0000 7f5646ffd700 1 -- 172.21.15.29:0/1681601058 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5648024ea0 con 0x7f5658076be0 2024-03-18T01:11:53.071 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:51.498+0000 7f565cfd1700 1 -- 172.21.15.29:0/1681601058 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5658112860 con 0x7f5658076be0 2024-03-18T01:11:53.071 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:51.499+0000 7f5646ffd700 1 -- 172.21.15.29:0/1681601058 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7f5648024780 con 0x7f5658076be0 2024-03-18T01:11:53.071 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:51.499+0000 7f565cfd1700 1 -- 172.21.15.29:0/1681601058 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f565805f5f0 con 0x7f5658076be0 2024-03-18T01:11:53.071 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:51.499+0000 7f5646ffd700 1 --2- 172.21.15.29:0/1681601058 >> [v2:172.21.15.29:6800/950856841,v1:172.21.15.29:6801/950856841] conn(0x7f5640036740 0x7f5640038bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:11:53.071 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:51.499+0000 7f5646ffd700 1 -- 172.21.15.29:0/1681601058 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f564804a6c0 con 0x7f5658076be0 2024-03-18T01:11:53.071 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:51.499+0000 7f5655d9b700 1 --2- 172.21.15.29:0/1681601058 >> [v2:172.21.15.29:6800/950856841,v1:172.21.15.29:6801/950856841] conn(0x7f5640036740 0x7f5640038bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:11:53.071 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:51.499+0000 7f5655d9b700 1 --2- 172.21.15.29:0/1681601058 >> [v2:172.21.15.29:6800/950856841,v1:172.21.15.29:6801/950856841] conn(0x7f5640036740 0x7f5640038bf0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f564c009990 tx=0x7f564c006e40 comp rx=0 tx=0).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:11:53.071 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:51.504+0000 7f5646ffd700 1 -- 172.21.15.29:0/1681601058 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175520 (secure 0 0 0) 0x7f564802c930 con 0x7f5658076be0 2024-03-18T01:11:53.072 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:51.748+0000 7f565cfd1700 1 -- 172.21.15.29:0/1681601058 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) v1 -- 0x7f565804bed0 con 0x7f5658076be0 2024-03-18T01:11:53.072 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:52.437+0000 7f5646ffd700 1 -- 172.21.15.29:0/1681601058 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mgrmap(e 9) v1 ==== 43381+0+0 (secure 0 0 0) 0x7f564802e440 con 0x7f5658076be0 2024-03-18T01:11:53.072 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:52.439+0000 7f5646ffd700 1 -- 172.21.15.29:0/1681601058 <== mon.0 v2:172.21.15.29:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) v1 ==== 88+0+0 (secure 0 0 0) 0x7f564800f930 con 0x7f5658076be0 2024-03-18T01:11:53.072 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:52.441+0000 7f565cfd1700 1 -- 172.21.15.29:0/1681601058 >> [v2:172.21.15.29:6800/950856841,v1:172.21.15.29:6801/950856841] conn(0x7f5640036740 msgr2=0x7f5640038bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:11:53.072 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:52.441+0000 7f565cfd1700 1 --2- 172.21.15.29:0/1681601058 >> [v2:172.21.15.29:6800/950856841,v1:172.21.15.29:6801/950856841] conn(0x7f5640036740 0x7f5640038bf0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f564c009990 tx=0x7f564c006e40 comp rx=0 tx=0).stop 2024-03-18T01:11:53.072 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:52.441+0000 7f565cfd1700 1 -- 172.21.15.29:0/1681601058 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5658076be0 msgr2=0x7f56581115b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:11:53.072 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:52.441+0000 7f565cfd1700 1 --2- 172.21.15.29:0/1681601058 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5658076be0 0x7f56581115b0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f5648006d40 tx=0x7f5648009250 comp rx=0 tx=0).stop 2024-03-18T01:11:53.072 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:52.441+0000 7f565cfd1700 1 -- 172.21.15.29:0/1681601058 shutdown_connections 2024-03-18T01:11:53.072 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:52.441+0000 7f565cfd1700 1 --2- 172.21.15.29:0/1681601058 >> [v2:172.21.15.29:6800/950856841,v1:172.21.15.29:6801/950856841] conn(0x7f5640036740 0x7f5640038bf0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:11:53.072 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:52.441+0000 7f565cfd1700 1 --2- 172.21.15.29:0/1681601058 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5658076be0 0x7f56581115b0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:11:53.072 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:52.441+0000 7f565cfd1700 1 -- 172.21.15.29:0/1681601058 >> 172.21.15.29:0/1681601058 conn(0x7f56580fcdc0 msgr2=0x7f56580fd960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:11:53.072 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:52.441+0000 7f565cfd1700 1 -- 172.21.15.29:0/1681601058 shutdown_connections 2024-03-18T01:11:53.072 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:52.441+0000 7f565cfd1700 1 -- 172.21.15.29:0/1681601058 wait complete. 2024-03-18T01:11:53.694 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:11:53 smithi029 ceph-mon[29331]: from='client.? 172.21.15.29:0/1681601058' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-03-18T01:11:53.694 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:11:53 smithi029 ceph-mon[29331]: mgrmap e9: smithi029.qviwhl(active, since 26s) 2024-03-18T01:11:54.358 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout { 2024-03-18T01:11:54.358 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-03-18T01:11:54.359 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-18T01:11:54.359 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "active_name": "smithi029.qviwhl", 2024-03-18T01:11:54.359 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-03-18T01:11:54.359 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout } 2024-03-18T01:11:54.359 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:53.487+0000 7f710bffd700 1 Processor -- start 2024-03-18T01:11:54.359 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:53.488+0000 7f710bffd700 1 -- start start 2024-03-18T01:11:54.359 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:53.488+0000 7f710bffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7104102710 0x7f7104102b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:11:54.359 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:53.488+0000 7f710bffd700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7104103060 con 0x7f7104102710 2024-03-18T01:11:54.359 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:53.488+0000 7f7109d99700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7104102710 0x7f7104102b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:11:54.359 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:53.488+0000 7f7109d99700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7104102710 0x7f7104102b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39150/0 (socket says 172.21.15.29:39150) 2024-03-18T01:11:54.359 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:53.488+0000 7f7109d99700 1 -- 172.21.15.29:0/1095360700 learned_addr learned my addr 172.21.15.29:0/1095360700 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-18T01:11:54.359 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:53.488+0000 7f7109d99700 1 -- 172.21.15.29:0/1095360700 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71041031a0 con 0x7f7104102710 2024-03-18T01:11:54.360 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:53.489+0000 7f7109d99700 1 --2- 172.21.15.29:0/1095360700 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7104102710 0x7f7104102b20 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f71000098e0 tx=0x7f7100009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea3ede0c61d05787 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:11:54.360 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:53.489+0000 7f7108d97700 1 -- 172.21.15.29:0/1095360700 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f710000b860 con 0x7f7104102710 2024-03-18T01:11:54.360 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:53.489+0000 7f7108d97700 1 -- 172.21.15.29:0/1095360700 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f710000be90 con 0x7f7104102710 2024-03-18T01:11:54.360 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:53.489+0000 7f7108d97700 1 -- 172.21.15.29:0/1095360700 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f71000103c0 con 0x7f7104102710 2024-03-18T01:11:54.360 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:53.490+0000 7f710bffd700 1 -- 172.21.15.29:0/1095360700 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7104102710 msgr2=0x7f7104102b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:11:54.360 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:53.490+0000 7f710bffd700 1 --2- 172.21.15.29:0/1095360700 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7104102710 0x7f7104102b20 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f71000098e0 tx=0x7f7100009250 comp rx=0 tx=0).stop 2024-03-18T01:11:54.360 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:53.490+0000 7f710bffd700 1 -- 172.21.15.29:0/1095360700 shutdown_connections 2024-03-18T01:11:54.360 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:53.490+0000 7f710bffd700 1 --2- 172.21.15.29:0/1095360700 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7104102710 0x7f7104102b20 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:11:54.360 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:53.490+0000 7f710bffd700 1 -- 172.21.15.29:0/1095360700 >> 172.21.15.29:0/1095360700 conn(0x7f71040fde50 msgr2=0x7f71041002a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:11:54.360 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:53.490+0000 7f710bffd700 1 -- 172.21.15.29:0/1095360700 shutdown_connections 2024-03-18T01:11:54.360 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:53.490+0000 7f710bffd700 1 -- 172.21.15.29:0/1095360700 wait complete. 2024-03-18T01:11:54.360 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:53.490+0000 7f710bffd700 1 Processor -- start 2024-03-18T01:11:54.360 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:53.491+0000 7f710bffd700 1 -- start start 2024-03-18T01:11:54.360 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:53.491+0000 7f710bffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7104102710 0x7f71041115c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:11:54.361 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:53.491+0000 7f710bffd700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7104111b00 con 0x7f7104102710 2024-03-18T01:11:54.361 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:53.491+0000 7f7109d99700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7104102710 0x7f71041115c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:11:54.361 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:53.491+0000 7f7109d99700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7104102710 0x7f71041115c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39162/0 (socket says 172.21.15.29:39162) 2024-03-18T01:11:54.361 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:53.491+0000 7f7109d99700 1 -- 172.21.15.29:0/1396118948 learned_addr learned my addr 172.21.15.29:0/1396118948 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-18T01:11:54.361 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:53.491+0000 7f7109d99700 1 -- 172.21.15.29:0/1396118948 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71000094f0 con 0x7f7104102710 2024-03-18T01:11:54.361 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:53.491+0000 7f7109d99700 1 --2- 172.21.15.29:0/1396118948 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7104102710 0x7f71041115c0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f71000090b0 tx=0x7f7100009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:11:54.361 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:53.491+0000 7f70f67fc700 1 -- 172.21.15.29:0/1396118948 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7100010840 con 0x7f7104102710 2024-03-18T01:11:54.361 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:53.492+0000 7f70f67fc700 1 -- 172.21.15.29:0/1396118948 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7100010e70 con 0x7f7104102710 2024-03-18T01:11:54.361 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:53.492+0000 7f710bffd700 1 -- 172.21.15.29:0/1396118948 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7104111d00 con 0x7f7104102710 2024-03-18T01:11:54.361 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:53.492+0000 7f70f67fc700 1 -- 172.21.15.29:0/1396118948 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7100024e40 con 0x7f7104102710 2024-03-18T01:11:54.361 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:53.492+0000 7f710bffd700 1 -- 172.21.15.29:0/1396118948 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7104112870 con 0x7f7104102710 2024-03-18T01:11:54.361 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:53.492+0000 7f710bffd700 1 -- 172.21.15.29:0/1396118948 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f710405f5f0 con 0x7f7104102710 2024-03-18T01:11:54.361 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:53.493+0000 7f70f67fc700 1 -- 172.21.15.29:0/1396118948 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 9) v1 ==== 43381+0+0 (secure 0 0 0) 0x7f71000109a0 con 0x7f7104102710 2024-03-18T01:11:54.361 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:53.493+0000 7f70f67fc700 1 --2- 172.21.15.29:0/1396118948 >> [v2:172.21.15.29:6800/950856841,v1:172.21.15.29:6801/950856841] conn(0x7f70f0036790 0x7f70f0038c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:11:54.361 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:53.493+0000 7f70f67fc700 1 -- 172.21.15.29:0/1396118948 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f710004a820 con 0x7f7104102710 2024-03-18T01:11:54.362 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:53.493+0000 7f7109598700 1 -- 172.21.15.29:0/1396118948 >> [v2:172.21.15.29:6800/950856841,v1:172.21.15.29:6801/950856841] conn(0x7f70f0036790 msgr2=0x7f70f0038c40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6800/950856841 2024-03-18T01:11:54.362 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:53.493+0000 7f7109598700 1 --2- 172.21.15.29:0/1396118948 >> [v2:172.21.15.29:6800/950856841,v1:172.21.15.29:6801/950856841] conn(0x7f70f0036790 0x7f70f0038c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T01:11:54.362 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:53.501+0000 7f70f67fc700 1 -- 172.21.15.29:0/1396118948 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175520 (secure 0 0 0) 0x7f710002c990 con 0x7f7104102710 2024-03-18T01:11:54.362 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:53.694+0000 7f7109598700 1 -- 172.21.15.29:0/1396118948 >> [v2:172.21.15.29:6800/950856841,v1:172.21.15.29:6801/950856841] conn(0x7f70f0036790 msgr2=0x7f70f0038c40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6800/950856841 2024-03-18T01:11:54.362 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:53.694+0000 7f7109598700 1 --2- 172.21.15.29:0/1396118948 >> [v2:172.21.15.29:6800/950856841,v1:172.21.15.29:6801/950856841] conn(0x7f70f0036790 0x7f70f0038c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T01:11:54.362 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:53.751+0000 7f710bffd700 1 -- 172.21.15.29:0/1396118948 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7f7104112bb0 con 0x7f7104102710 2024-03-18T01:11:54.362 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:53.751+0000 7f70f67fc700 1 -- 172.21.15.29:0/1396118948 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) v1 ==== 56+0+103 (secure 0 0 0) 0x7f710002c7b0 con 0x7f7104102710 2024-03-18T01:11:54.362 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:53.754+0000 7f710bffd700 1 -- 172.21.15.29:0/1396118948 >> [v2:172.21.15.29:6800/950856841,v1:172.21.15.29:6801/950856841] conn(0x7f70f0036790 msgr2=0x7f70f0038c40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T01:11:54.362 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:53.754+0000 7f710bffd700 1 --2- 172.21.15.29:0/1396118948 >> [v2:172.21.15.29:6800/950856841,v1:172.21.15.29:6801/950856841] conn(0x7f70f0036790 0x7f70f0038c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:11:54.362 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:53.754+0000 7f710bffd700 1 -- 172.21.15.29:0/1396118948 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7104102710 msgr2=0x7f71041115c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:11:54.362 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:53.754+0000 7f710bffd700 1 --2- 172.21.15.29:0/1396118948 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7104102710 0x7f71041115c0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f71000090b0 tx=0x7f7100009250 comp rx=0 tx=0).stop 2024-03-18T01:11:54.362 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:53.754+0000 7f710bffd700 1 -- 172.21.15.29:0/1396118948 shutdown_connections 2024-03-18T01:11:54.362 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:53.755+0000 7f710bffd700 1 --2- 172.21.15.29:0/1396118948 >> [v2:172.21.15.29:6800/950856841,v1:172.21.15.29:6801/950856841] conn(0x7f70f0036790 0x7f70f0038c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:11:54.362 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:53.755+0000 7f710bffd700 1 --2- 172.21.15.29:0/1396118948 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7104102710 0x7f71041115c0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:11:54.362 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:53.755+0000 7f710bffd700 1 -- 172.21.15.29:0/1396118948 >> 172.21.15.29:0/1396118948 conn(0x7f71040fde50 msgr2=0x7f71040fea60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:11:54.363 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:53.755+0000 7f710bffd700 1 -- 172.21.15.29:0/1396118948 shutdown_connections 2024-03-18T01:11:54.363 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:53.755+0000 7f710bffd700 1 -- 172.21.15.29:0/1396118948 wait complete. 2024-03-18T01:11:54.363 INFO:teuthology.orchestra.run.smithi029.stdout:Waiting for the mgr to restart... 2024-03-18T01:11:54.363 INFO:teuthology.orchestra.run.smithi029.stdout:Waiting for mgr epoch 9... 2024-03-18T01:11:54.617 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:11:54 smithi029 ceph-mon[29331]: from='client.? 172.21.15.29:0/1396118948' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-18T01:11:59.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:11:59 smithi029 ceph-mon[29331]: Active manager daemon smithi029.qviwhl restarted 2024-03-18T01:11:59.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:11:59 smithi029 ceph-mon[29331]: Activating manager daemon smithi029.qviwhl 2024-03-18T01:11:59.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:11:59 smithi029 ceph-mon[29331]: osdmap e3: 0 total, 0 up, 0 in 2024-03-18T01:11:59.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:11:59 smithi029 ceph-mon[29331]: mgrmap e10: smithi029.qviwhl(active, starting, since 0.00320888s) 2024-03-18T01:11:59.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:11:59 smithi029 ceph-mon[29331]: from='mgr.14158 172.21.15.29:0/4059439045' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "mon metadata", "id": "smithi029"}]: dispatch 2024-03-18T01:11:59.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:11:59 smithi029 ceph-mon[29331]: from='mgr.14158 172.21.15.29:0/4059439045' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "mgr metadata", "who": "smithi029.qviwhl", "id": "smithi029.qviwhl"}]: dispatch 2024-03-18T01:11:59.570 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:11:59 smithi029 ceph-mon[29331]: from='mgr.14158 172.21.15.29:0/4059439045' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-18T01:11:59.570 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:11:59 smithi029 ceph-mon[29331]: from='mgr.14158 172.21.15.29:0/4059439045' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-18T01:11:59.570 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:11:59 smithi029 ceph-mon[29331]: from='mgr.14158 172.21.15.29:0/4059439045' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-18T01:11:59.570 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:11:59 smithi029 ceph-mon[29331]: Manager daemon smithi029.qviwhl is now available 2024-03-18T01:11:59.570 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:11:59 smithi029 ceph-mon[29331]: from='mgr.14158 172.21.15.29:0/4059439045' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T01:11:59.570 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:11:59 smithi029 ceph-mon[29331]: from='mgr.14158 172.21.15.29:0/4059439045' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T01:12:00.502 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:12:00 smithi029 ceph-mon[29331]: from='mgr.14158 172.21.15.29:0/4059439045' entity='mgr.smithi029.qviwhl' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi029.qviwhl/mirror_snapshot_schedule"}]: dispatch 2024-03-18T01:12:00.502 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:12:00 smithi029 ceph-mon[29331]: from='mgr.14158 172.21.15.29:0/4059439045' entity='mgr.smithi029.qviwhl' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi029.qviwhl/trash_purge_schedule"}]: dispatch 2024-03-18T01:12:00.502 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:12:00 smithi029 ceph-mon[29331]: [18/Mar/2024:01:11:59] ENGINE Bus STARTING 2024-03-18T01:12:00.502 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:12:00 smithi029 ceph-mon[29331]: from='mgr.14158 172.21.15.29:0/4059439045' entity='mgr.smithi029.qviwhl' 2024-03-18T01:12:00.502 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:12:00 smithi029 ceph-mon[29331]: mgrmap e11: smithi029.qviwhl(active, since 1.00525s) 2024-03-18T01:12:00.790 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout { 2024-03-18T01:12:00.790 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-03-18T01:12:00.790 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "initialized": true 2024-03-18T01:12:00.790 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout } 2024-03-18T01:12:00.790 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:54.771+0000 7fe2b47f3700 1 Processor -- start 2024-03-18T01:12:00.790 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:54.771+0000 7fe2b47f3700 1 -- start start 2024-03-18T01:12:00.790 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:54.772+0000 7fe2b47f3700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe2ac078cc0 0x7fe2ac0771c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:12:00.790 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:54.772+0000 7fe2b47f3700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2ac0790d0 con 0x7fe2ac078cc0 2024-03-18T01:12:00.790 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:54.772+0000 7fe2b258f700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe2ac078cc0 0x7fe2ac0771c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:12:00.791 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:54.772+0000 7fe2b258f700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe2ac078cc0 0x7fe2ac0771c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39168/0 (socket says 172.21.15.29:39168) 2024-03-18T01:12:00.791 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:54.772+0000 7fe2b258f700 1 -- 172.21.15.29:0/2891221677 learned_addr learned my addr 172.21.15.29:0/2891221677 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-18T01:12:00.791 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:54.772+0000 7fe2b258f700 1 -- 172.21.15.29:0/2891221677 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2ac077700 con 0x7fe2ac078cc0 2024-03-18T01:12:00.791 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:54.772+0000 7fe2b258f700 1 --2- 172.21.15.29:0/2891221677 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe2ac078cc0 0x7fe2ac0771c0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fe2a00098e0 tx=0x7fe2a0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c3579f9dce0514e3 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:12:00.791 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:54.773+0000 7fe2b158d700 1 -- 172.21.15.29:0/2891221677 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe2a000b860 con 0x7fe2ac078cc0 2024-03-18T01:12:00.791 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:54.773+0000 7fe2b158d700 1 -- 172.21.15.29:0/2891221677 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe2a000be90 con 0x7fe2ac078cc0 2024-03-18T01:12:00.791 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:54.773+0000 7fe2b158d700 1 -- 172.21.15.29:0/2891221677 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe2a00103c0 con 0x7fe2ac078cc0 2024-03-18T01:12:00.791 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:54.773+0000 7fe2b47f3700 1 -- 172.21.15.29:0/2891221677 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe2ac078cc0 msgr2=0x7fe2ac0771c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:12:00.791 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:54.773+0000 7fe2b47f3700 1 --2- 172.21.15.29:0/2891221677 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe2ac078cc0 0x7fe2ac0771c0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fe2a00098e0 tx=0x7fe2a0009250 comp rx=0 tx=0).stop 2024-03-18T01:12:00.791 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:54.773+0000 7fe2b47f3700 1 -- 172.21.15.29:0/2891221677 shutdown_connections 2024-03-18T01:12:00.791 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:54.773+0000 7fe2b47f3700 1 --2- 172.21.15.29:0/2891221677 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe2ac078cc0 0x7fe2ac0771c0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:12:00.791 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:54.773+0000 7fe2b47f3700 1 -- 172.21.15.29:0/2891221677 >> 172.21.15.29:0/2891221677 conn(0x7fe2ac0fdf10 msgr2=0x7fe2ac100320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:12:00.791 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:54.773+0000 7fe2b47f3700 1 -- 172.21.15.29:0/2891221677 shutdown_connections 2024-03-18T01:12:00.791 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:54.774+0000 7fe2b47f3700 1 -- 172.21.15.29:0/2891221677 wait complete. 2024-03-18T01:12:00.791 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:54.774+0000 7fe2b47f3700 1 Processor -- start 2024-03-18T01:12:00.792 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:54.774+0000 7fe2b47f3700 1 -- start start 2024-03-18T01:12:00.792 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:54.774+0000 7fe2b47f3700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe2ac078cc0 0x7fe2ac1137f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:12:00.792 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:54.774+0000 7fe2b47f3700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2ac113d30 con 0x7fe2ac078cc0 2024-03-18T01:12:00.792 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:54.774+0000 7fe2b258f700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe2ac078cc0 0x7fe2ac1137f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:12:00.792 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:54.774+0000 7fe2b258f700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe2ac078cc0 0x7fe2ac1137f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39172/0 (socket says 172.21.15.29:39172) 2024-03-18T01:12:00.792 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:54.774+0000 7fe2b258f700 1 -- 172.21.15.29:0/2739413137 learned_addr learned my addr 172.21.15.29:0/2739413137 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-18T01:12:00.792 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:54.775+0000 7fe2b258f700 1 -- 172.21.15.29:0/2739413137 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2a00094f0 con 0x7fe2ac078cc0 2024-03-18T01:12:00.792 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:54.775+0000 7fe2b258f700 1 --2- 172.21.15.29:0/2739413137 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe2ac078cc0 0x7fe2ac1137f0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fe2a00094c0 tx=0x7fe2a0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:12:00.792 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:54.775+0000 7fe29effd700 1 -- 172.21.15.29:0/2739413137 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe2a0010890 con 0x7fe2ac078cc0 2024-03-18T01:12:00.792 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:54.775+0000 7fe29effd700 1 -- 172.21.15.29:0/2739413137 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe2a0010ec0 con 0x7fe2ac078cc0 2024-03-18T01:12:00.792 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:54.775+0000 7fe2b47f3700 1 -- 172.21.15.29:0/2739413137 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2ac113f30 con 0x7fe2ac078cc0 2024-03-18T01:12:00.792 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:54.775+0000 7fe29effd700 1 -- 172.21.15.29:0/2739413137 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe2a00243f0 con 0x7fe2ac078cc0 2024-03-18T01:12:00.792 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:54.775+0000 7fe2b47f3700 1 -- 172.21.15.29:0/2739413137 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2ac114aa0 con 0x7fe2ac078cc0 2024-03-18T01:12:00.792 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:54.776+0000 7fe29effd700 1 -- 172.21.15.29:0/2739413137 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 9) v1 ==== 43381+0+0 (secure 0 0 0) 0x7fe2a002d3f0 con 0x7fe2ac078cc0 2024-03-18T01:12:00.793 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:54.776+0000 7fe29effd700 1 --2- 172.21.15.29:0/2739413137 >> [v2:172.21.15.29:6800/950856841,v1:172.21.15.29:6801/950856841] conn(0x7fe2980367e0 0x7fe298038c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:12:00.793 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:54.776+0000 7fe29effd700 1 -- 172.21.15.29:0/2739413137 --> [v2:172.21.15.29:6800/950856841,v1:172.21.15.29:6801/950856841] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe2980393a0 con 0x7fe2980367e0 2024-03-18T01:12:00.793 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:54.776+0000 7fe29effd700 1 -- 172.21.15.29:0/2739413137 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7fe2a004a2f0 con 0x7fe2ac078cc0 2024-03-18T01:12:00.793 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:54.776+0000 7fe2b1d8e700 1 -- 172.21.15.29:0/2739413137 >> [v2:172.21.15.29:6800/950856841,v1:172.21.15.29:6801/950856841] conn(0x7fe2980367e0 msgr2=0x7fe298038c90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6800/950856841 2024-03-18T01:12:00.793 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:54.776+0000 7fe2b1d8e700 1 --2- 172.21.15.29:0/2739413137 >> [v2:172.21.15.29:6800/950856841,v1:172.21.15.29:6801/950856841] conn(0x7fe2980367e0 0x7fe298038c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T01:12:00.793 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:54.977+0000 7fe2b1d8e700 1 -- 172.21.15.29:0/2739413137 >> [v2:172.21.15.29:6800/950856841,v1:172.21.15.29:6801/950856841] conn(0x7fe2980367e0 msgr2=0x7fe298038c90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6800/950856841 2024-03-18T01:12:00.793 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:54.977+0000 7fe2b1d8e700 1 --2- 172.21.15.29:0/2739413137 >> [v2:172.21.15.29:6800/950856841,v1:172.21.15.29:6801/950856841] conn(0x7fe2980367e0 0x7fe298038c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T01:12:00.793 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:55.378+0000 7fe2b1d8e700 1 -- 172.21.15.29:0/2739413137 >> [v2:172.21.15.29:6800/950856841,v1:172.21.15.29:6801/950856841] conn(0x7fe2980367e0 msgr2=0x7fe298038c90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6800/950856841 2024-03-18T01:12:00.793 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:55.378+0000 7fe2b1d8e700 1 --2- 172.21.15.29:0/2739413137 >> [v2:172.21.15.29:6800/950856841,v1:172.21.15.29:6801/950856841] conn(0x7fe2980367e0 0x7fe298038c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-18T01:12:00.793 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:56.179+0000 7fe2b1d8e700 1 -- 172.21.15.29:0/2739413137 >> [v2:172.21.15.29:6800/950856841,v1:172.21.15.29:6801/950856841] conn(0x7fe2980367e0 msgr2=0x7fe298038c90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6800/950856841 2024-03-18T01:12:00.793 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:56.179+0000 7fe2b1d8e700 1 --2- 172.21.15.29:0/2739413137 >> [v2:172.21.15.29:6800/950856841,v1:172.21.15.29:6801/950856841] conn(0x7fe2980367e0 0x7fe298038c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-18T01:12:00.793 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:57.779+0000 7fe2b1d8e700 1 -- 172.21.15.29:0/2739413137 >> [v2:172.21.15.29:6800/950856841,v1:172.21.15.29:6801/950856841] conn(0x7fe2980367e0 msgr2=0x7fe298038c90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6800/950856841 2024-03-18T01:12:00.793 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:57.779+0000 7fe2b1d8e700 1 --2- 172.21.15.29:0/2739413137 >> [v2:172.21.15.29:6800/950856841,v1:172.21.15.29:6801/950856841] conn(0x7fe2980367e0 0x7fe298038c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-03-18T01:12:00.793 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:59.170+0000 7fe29effd700 1 -- 172.21.15.29:0/2739413137 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mgrmap(e 10) v1 ==== 43148+0+0 (secure 0 0 0) 0x7fe2a001a7e0 con 0x7fe2ac078cc0 2024-03-18T01:12:00.794 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:59.170+0000 7fe29effd700 1 -- 172.21.15.29:0/2739413137 >> [v2:172.21.15.29:6800/950856841,v1:172.21.15.29:6801/950856841] conn(0x7fe2980367e0 msgr2=0x7fe298038c90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T01:12:00.794 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:11:59.170+0000 7fe29effd700 1 --2- 172.21.15.29:0/2739413137 >> [v2:172.21.15.29:6800/950856841,v1:172.21.15.29:6801/950856841] conn(0x7fe2980367e0 0x7fe298038c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:12:00.794 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:00.171+0000 7fe29effd700 1 -- 172.21.15.29:0/2739413137 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mgrmap(e 11) v1 ==== 43275+0+0 (secure 0 0 0) 0x7fe2a002dac0 con 0x7fe2ac078cc0 2024-03-18T01:12:00.794 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:00.171+0000 7fe29effd700 1 --2- 172.21.15.29:0/2739413137 >> [v2:172.21.15.29:6800/2687730237,v1:172.21.15.29:6801/2687730237] conn(0x7fe29803a070 0x7fe29803c450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:12:00.794 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:00.171+0000 7fe29effd700 1 -- 172.21.15.29:0/2739413137 --> [v2:172.21.15.29:6800/2687730237,v1:172.21.15.29:6801/2687730237] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe2980393a0 con 0x7fe29803a070 2024-03-18T01:12:00.794 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:00.172+0000 7fe2b1d8e700 1 --2- 172.21.15.29:0/2739413137 >> [v2:172.21.15.29:6800/2687730237,v1:172.21.15.29:6801/2687730237] conn(0x7fe29803a070 0x7fe29803c450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:12:00.794 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:00.172+0000 7fe2b1d8e700 1 --2- 172.21.15.29:0/2739413137 >> [v2:172.21.15.29:6800/2687730237,v1:172.21.15.29:6801/2687730237] conn(0x7fe29803a070 0x7fe29803c450 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fe2a80039d0 tx=0x7fe2a80092f0 comp rx=0 tx=0).ready entity=mgr.14158 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:12:00.794 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:00.173+0000 7fe29effd700 1 -- 172.21.15.29:0/2739413137 <== mgr.14158 v2:172.21.15.29:6800/2687730237 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6549 (secure 0 0 0) 0x7fe2980393a0 con 0x7fe29803a070 2024-03-18T01:12:00.794 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:00.179+0000 7fe2b47f3700 1 -- 172.21.15.29:0/2739413137 --> [v2:172.21.15.29:6800/2687730237,v1:172.21.15.29:6801/2687730237] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7fe2ac1143c0 con 0x7fe29803a070 2024-03-18T01:12:00.794 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:00.179+0000 7fe29effd700 1 -- 172.21.15.29:0/2739413137 <== mgr.14158 v2:172.21.15.29:6800/2687730237 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+52 (secure 0 0 0) 0x7fe2ac1143c0 con 0x7fe29803a070 2024-03-18T01:12:00.794 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:00.179+0000 7fe2b47f3700 1 -- 172.21.15.29:0/2739413137 >> [v2:172.21.15.29:6800/2687730237,v1:172.21.15.29:6801/2687730237] conn(0x7fe29803a070 msgr2=0x7fe29803c450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:12:00.794 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:00.179+0000 7fe2b47f3700 1 --2- 172.21.15.29:0/2739413137 >> [v2:172.21.15.29:6800/2687730237,v1:172.21.15.29:6801/2687730237] conn(0x7fe29803a070 0x7fe29803c450 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fe2a80039d0 tx=0x7fe2a80092f0 comp rx=0 tx=0).stop 2024-03-18T01:12:00.794 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:00.179+0000 7fe2b47f3700 1 -- 172.21.15.29:0/2739413137 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe2ac078cc0 msgr2=0x7fe2ac1137f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:12:00.794 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:00.179+0000 7fe2b47f3700 1 --2- 172.21.15.29:0/2739413137 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe2ac078cc0 0x7fe2ac1137f0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fe2a00094c0 tx=0x7fe2a0009250 comp rx=0 tx=0).stop 2024-03-18T01:12:00.795 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:00.180+0000 7fe2b47f3700 1 -- 172.21.15.29:0/2739413137 shutdown_connections 2024-03-18T01:12:00.795 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:00.180+0000 7fe2b47f3700 1 --2- 172.21.15.29:0/2739413137 >> [v2:172.21.15.29:6800/2687730237,v1:172.21.15.29:6801/2687730237] conn(0x7fe29803a070 0x7fe29803c450 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:12:00.795 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:00.180+0000 7fe2b47f3700 1 --2- 172.21.15.29:0/2739413137 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe2ac078cc0 0x7fe2ac1137f0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:12:00.795 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:00.180+0000 7fe2b47f3700 1 --2- 172.21.15.29:0/2739413137 >> [v2:172.21.15.29:6800/950856841,v1:172.21.15.29:6801/950856841] conn(0x7fe2980367e0 0x7fe298038c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:12:00.795 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:00.180+0000 7fe2b47f3700 1 -- 172.21.15.29:0/2739413137 >> 172.21.15.29:0/2739413137 conn(0x7fe2ac0fdf10 msgr2=0x7fe2ac0fead0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:12:00.795 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:00.180+0000 7fe2b47f3700 1 -- 172.21.15.29:0/2739413137 shutdown_connections 2024-03-18T01:12:00.795 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:00.180+0000 7fe2b47f3700 1 -- 172.21.15.29:0/2739413137 wait complete. 2024-03-18T01:12:00.795 INFO:teuthology.orchestra.run.smithi029.stdout:mgr epoch 9 is available 2024-03-18T01:12:00.795 INFO:teuthology.orchestra.run.smithi029.stdout:Generating a dashboard self-signed certificate... 2024-03-18T01:12:01.483 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:12:01 smithi029 ceph-mon[29331]: [18/Mar/2024:01:11:59] ENGINE Serving on https://172.21.15.29:7150 2024-03-18T01:12:01.484 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:12:01 smithi029 ceph-mon[29331]: [18/Mar/2024:01:11:59] ENGINE Bus STARTED 2024-03-18T01:12:01.484 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:12:01 smithi029 ceph-mon[29331]: from='client.14162 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-03-18T01:12:01.484 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:12:01 smithi029 ceph-mon[29331]: from='client.14162 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-03-18T01:12:01.484 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:12:01 smithi029 ceph-mon[29331]: from='mgr.14158 172.21.15.29:0/4059439045' entity='mgr.smithi029.qviwhl' 2024-03-18T01:12:02.154 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-03-18T01:12:02.154 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:01.274+0000 7f297f379700 1 Processor -- start 2024-03-18T01:12:02.154 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:01.274+0000 7f297f379700 1 -- start start 2024-03-18T01:12:02.154 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:01.274+0000 7f297f379700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2978102730 0x7f2978102b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:12:02.154 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:01.274+0000 7f297f379700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2978103080 con 0x7f2978102730 2024-03-18T01:12:02.154 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:01.274+0000 7f297d115700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2978102730 0x7f2978102b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:12:02.155 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:01.274+0000 7f297d115700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2978102730 0x7f2978102b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39256/0 (socket says 172.21.15.29:39256) 2024-03-18T01:12:02.155 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:01.274+0000 7f297d115700 1 -- 172.21.15.29:0/2636305040 learned_addr learned my addr 172.21.15.29:0/2636305040 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-18T01:12:02.155 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:01.275+0000 7f297d115700 1 -- 172.21.15.29:0/2636305040 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29781031c0 con 0x7f2978102730 2024-03-18T01:12:02.155 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:01.275+0000 7f297d115700 1 --2- 172.21.15.29:0/2636305040 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2978102730 0x7f2978102b40 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f29740098e0 tx=0x7f2974009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d40cbd5a509775a9 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:12:02.155 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:01.275+0000 7f296ffff700 1 -- 172.21.15.29:0/2636305040 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f297400b860 con 0x7f2978102730 2024-03-18T01:12:02.155 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:01.275+0000 7f296ffff700 1 -- 172.21.15.29:0/2636305040 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f297400be90 con 0x7f2978102730 2024-03-18T01:12:02.155 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:01.275+0000 7f296ffff700 1 -- 172.21.15.29:0/2636305040 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f29740103c0 con 0x7f2978102730 2024-03-18T01:12:02.155 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:01.275+0000 7f297f379700 1 -- 172.21.15.29:0/2636305040 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2978102730 msgr2=0x7f2978102b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:12:02.155 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:01.275+0000 7f297f379700 1 --2- 172.21.15.29:0/2636305040 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2978102730 0x7f2978102b40 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f29740098e0 tx=0x7f2974009250 comp rx=0 tx=0).stop 2024-03-18T01:12:02.155 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:01.275+0000 7f297f379700 1 -- 172.21.15.29:0/2636305040 shutdown_connections 2024-03-18T01:12:02.155 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:01.275+0000 7f297f379700 1 --2- 172.21.15.29:0/2636305040 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2978102730 0x7f2978102b40 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:12:02.155 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:01.275+0000 7f297f379700 1 -- 172.21.15.29:0/2636305040 >> 172.21.15.29:0/2636305040 conn(0x7f29780fdf10 msgr2=0x7f2978100320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:12:02.155 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:01.276+0000 7f297f379700 1 -- 172.21.15.29:0/2636305040 shutdown_connections 2024-03-18T01:12:02.155 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:01.276+0000 7f297f379700 1 -- 172.21.15.29:0/2636305040 wait complete. 2024-03-18T01:12:02.156 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:01.276+0000 7f297f379700 1 Processor -- start 2024-03-18T01:12:02.156 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:01.276+0000 7f297f379700 1 -- start start 2024-03-18T01:12:02.156 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:01.276+0000 7f297f379700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2978102730 0x7f2978111630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:12:02.156 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:01.276+0000 7f297f379700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2978111b70 con 0x7f2978102730 2024-03-18T01:12:02.156 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:01.276+0000 7f297d115700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2978102730 0x7f2978111630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:12:02.156 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:01.276+0000 7f297d115700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2978102730 0x7f2978111630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39268/0 (socket says 172.21.15.29:39268) 2024-03-18T01:12:02.156 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:01.276+0000 7f297d115700 1 -- 172.21.15.29:0/1904332664 learned_addr learned my addr 172.21.15.29:0/1904332664 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-18T01:12:02.156 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:01.277+0000 7f297d115700 1 -- 172.21.15.29:0/1904332664 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29740094f0 con 0x7f2978102730 2024-03-18T01:12:02.156 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:01.277+0000 7f297d115700 1 --2- 172.21.15.29:0/1904332664 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2978102730 0x7f2978111630 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f29740094c0 tx=0x7f2974009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:12:02.156 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:01.277+0000 7f296dffb700 1 -- 172.21.15.29:0/1904332664 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2974010680 con 0x7f2978102730 2024-03-18T01:12:02.156 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:01.277+0000 7f296dffb700 1 -- 172.21.15.29:0/1904332664 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2974010cb0 con 0x7f2978102730 2024-03-18T01:12:02.156 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:01.277+0000 7f296dffb700 1 -- 172.21.15.29:0/1904332664 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f297401ad40 con 0x7f2978102730 2024-03-18T01:12:02.157 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:01.277+0000 7f297f379700 1 -- 172.21.15.29:0/1904332664 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2978111d70 con 0x7f2978102730 2024-03-18T01:12:02.157 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:01.277+0000 7f297f379700 1 -- 172.21.15.29:0/1904332664 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29781128e0 con 0x7f2978102730 2024-03-18T01:12:02.157 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:01.278+0000 7f296dffb700 1 -- 172.21.15.29:0/1904332664 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 11) v1 ==== 43275+0+0 (secure 0 0 0) 0x7f29740107e0 con 0x7f2978102730 2024-03-18T01:12:02.157 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:01.278+0000 7f296dffb700 1 --2- 172.21.15.29:0/1904332664 >> [v2:172.21.15.29:6800/2687730237,v1:172.21.15.29:6801/2687730237] conn(0x7f29640366b0 0x7f2964038b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:12:02.157 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:01.278+0000 7f296dffb700 1 -- 172.21.15.29:0/1904332664 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f297404a220 con 0x7f2978102730 2024-03-18T01:12:02.157 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:01.278+0000 7f297f379700 1 -- 172.21.15.29:0/1904332664 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f297805f5f0 con 0x7f2978102730 2024-03-18T01:12:02.157 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:01.278+0000 7f297c914700 1 --2- 172.21.15.29:0/1904332664 >> [v2:172.21.15.29:6800/2687730237,v1:172.21.15.29:6801/2687730237] conn(0x7f29640366b0 0x7f2964038b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:12:02.157 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:01.279+0000 7f297c914700 1 --2- 172.21.15.29:0/1904332664 >> [v2:172.21.15.29:6800/2687730237,v1:172.21.15.29:6801/2687730237] conn(0x7f29640366b0 0x7f2964038b60 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f2968009990 tx=0x7f2968006e40 comp rx=0 tx=0).ready entity=mgr.14158 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:12:02.157 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:01.283+0000 7f296dffb700 1 -- 172.21.15.29:0/1904332664 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175520 (secure 0 0 0) 0x7f2974020070 con 0x7f2978102730 2024-03-18T01:12:02.157 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:01.482+0000 7f297f379700 1 -- 172.21.15.29:0/1904332664 --> [v2:172.21.15.29:6800/2687730237,v1:172.21.15.29:6801/2687730237] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) v1 -- 0x7f29780ff2d0 con 0x7f29640366b0 2024-03-18T01:12:02.157 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:01.510+0000 7f296dffb700 1 -- 172.21.15.29:0/1904332664 <== mgr.14158 v2:172.21.15.29:6800/2687730237 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+32 (secure 0 0 0) 0x7f29780ff2d0 con 0x7f29640366b0 2024-03-18T01:12:02.157 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:01.512+0000 7f297f379700 1 -- 172.21.15.29:0/1904332664 >> [v2:172.21.15.29:6800/2687730237,v1:172.21.15.29:6801/2687730237] conn(0x7f29640366b0 msgr2=0x7f2964038b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:12:02.157 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:01.512+0000 7f297f379700 1 --2- 172.21.15.29:0/1904332664 >> [v2:172.21.15.29:6800/2687730237,v1:172.21.15.29:6801/2687730237] conn(0x7f29640366b0 0x7f2964038b60 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f2968009990 tx=0x7f2968006e40 comp rx=0 tx=0).stop 2024-03-18T01:12:02.157 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:01.512+0000 7f297f379700 1 -- 172.21.15.29:0/1904332664 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2978102730 msgr2=0x7f2978111630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:12:02.157 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:01.512+0000 7f297f379700 1 --2- 172.21.15.29:0/1904332664 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2978102730 0x7f2978111630 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f29740094c0 tx=0x7f2974009250 comp rx=0 tx=0).stop 2024-03-18T01:12:02.158 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:01.512+0000 7f297f379700 1 -- 172.21.15.29:0/1904332664 shutdown_connections 2024-03-18T01:12:02.158 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:01.512+0000 7f297f379700 1 --2- 172.21.15.29:0/1904332664 >> [v2:172.21.15.29:6800/2687730237,v1:172.21.15.29:6801/2687730237] conn(0x7f29640366b0 0x7f2964038b60 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:12:02.158 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:01.512+0000 7f297f379700 1 --2- 172.21.15.29:0/1904332664 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2978102730 0x7f2978111630 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:12:02.158 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:01.512+0000 7f297f379700 1 -- 172.21.15.29:0/1904332664 >> 172.21.15.29:0/1904332664 conn(0x7f29780fdf10 msgr2=0x7f29780feb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:12:02.158 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:01.512+0000 7f297f379700 1 -- 172.21.15.29:0/1904332664 shutdown_connections 2024-03-18T01:12:02.158 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:01.512+0000 7f297f379700 1 -- 172.21.15.29:0/1904332664 wait complete. 2024-03-18T01:12:02.158 INFO:teuthology.orchestra.run.smithi029.stdout:Creating initial admin user... 2024-03-18T01:12:02.488 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:12:02 smithi029 ceph-mon[29331]: from='client.14170 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T01:12:02.488 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:12:02 smithi029 ceph-mon[29331]: from='mgr.14158 172.21.15.29:0/4059439045' entity='mgr.smithi029.qviwhl' 2024-03-18T01:12:02.488 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:12:02 smithi029 ceph-mon[29331]: from='mgr.14158 172.21.15.29:0/4059439045' entity='mgr.smithi029.qviwhl' 2024-03-18T01:12:02.488 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:12:02 smithi029 ceph-mon[29331]: from='mgr.14158 172.21.15.29:0/4059439045' entity='mgr.smithi029.qviwhl' 2024-03-18T01:12:02.488 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:12:02 smithi029 ceph-mon[29331]: mgrmap e12: smithi029.qviwhl(active, since 2s) 2024-03-18T01:12:03.942 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$lD86M4mGOrbF7Sq.k25Ncu8AaxfFWZ2SjvhlOxPRdWO8aReaOPwN6", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1710724323, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-03-18T01:12:03.942 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:02.644+0000 7f6596f6e700 1 Processor -- start 2024-03-18T01:12:03.942 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:02.645+0000 7f6596f6e700 1 -- start start 2024-03-18T01:12:03.942 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:02.645+0000 7f6596f6e700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6590102730 0x7f6590102b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:12:03.942 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:02.645+0000 7f6596f6e700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6590103080 con 0x7f6590102730 2024-03-18T01:12:03.943 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:02.645+0000 7f6594d0a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6590102730 0x7f6590102b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:12:03.943 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:02.645+0000 7f6594d0a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6590102730 0x7f6590102b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:55454/0 (socket says 172.21.15.29:55454) 2024-03-18T01:12:03.943 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:02.645+0000 7f6594d0a700 1 -- 172.21.15.29:0/806423829 learned_addr learned my addr 172.21.15.29:0/806423829 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-18T01:12:03.943 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:02.645+0000 7f6594d0a700 1 -- 172.21.15.29:0/806423829 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65901031c0 con 0x7f6590102730 2024-03-18T01:12:03.943 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:02.645+0000 7f6594d0a700 1 --2- 172.21.15.29:0/806423829 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6590102730 0x7f6590102b40 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f65840098e0 tx=0x7f6584009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a54b43cd4b9686c server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:12:03.943 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:02.645+0000 7f658f7fe700 1 -- 172.21.15.29:0/806423829 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f658400b860 con 0x7f6590102730 2024-03-18T01:12:03.944 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:02.646+0000 7f658f7fe700 1 -- 172.21.15.29:0/806423829 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f658400be90 con 0x7f6590102730 2024-03-18T01:12:03.944 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:02.646+0000 7f658f7fe700 1 -- 172.21.15.29:0/806423829 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f65840103c0 con 0x7f6590102730 2024-03-18T01:12:03.944 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:02.646+0000 7f6596f6e700 1 -- 172.21.15.29:0/806423829 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6590102730 msgr2=0x7f6590102b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:12:03.944 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:02.646+0000 7f6596f6e700 1 --2- 172.21.15.29:0/806423829 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6590102730 0x7f6590102b40 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f65840098e0 tx=0x7f6584009250 comp rx=0 tx=0).stop 2024-03-18T01:12:03.944 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:02.646+0000 7f6596f6e700 1 -- 172.21.15.29:0/806423829 shutdown_connections 2024-03-18T01:12:03.945 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:02.646+0000 7f6596f6e700 1 --2- 172.21.15.29:0/806423829 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6590102730 0x7f6590102b40 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:12:03.945 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:02.646+0000 7f6596f6e700 1 -- 172.21.15.29:0/806423829 >> 172.21.15.29:0/806423829 conn(0x7f65900fdf10 msgr2=0x7f6590100320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:12:03.945 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:02.646+0000 7f6596f6e700 1 -- 172.21.15.29:0/806423829 shutdown_connections 2024-03-18T01:12:03.945 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:02.646+0000 7f6596f6e700 1 -- 172.21.15.29:0/806423829 wait complete. 2024-03-18T01:12:03.945 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:02.647+0000 7f6596f6e700 1 Processor -- start 2024-03-18T01:12:03.946 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:02.647+0000 7f6596f6e700 1 -- start start 2024-03-18T01:12:03.946 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:02.647+0000 7f6596f6e700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6590102730 0x7f6590078bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:12:03.946 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:02.647+0000 7f6596f6e700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6590079110 con 0x7f6590102730 2024-03-18T01:12:03.946 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:02.647+0000 7f6594d0a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6590102730 0x7f6590078bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:12:03.946 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:02.647+0000 7f6594d0a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6590102730 0x7f6590078bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:55458/0 (socket says 172.21.15.29:55458) 2024-03-18T01:12:03.946 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:02.647+0000 7f6594d0a700 1 -- 172.21.15.29:0/771528107 learned_addr learned my addr 172.21.15.29:0/771528107 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-18T01:12:03.947 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:02.647+0000 7f6594d0a700 1 -- 172.21.15.29:0/771528107 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65840094f0 con 0x7f6590102730 2024-03-18T01:12:03.947 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:02.647+0000 7f6594d0a700 1 --2- 172.21.15.29:0/771528107 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6590102730 0x7f6590078bd0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f65840094c0 tx=0x7f6584009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:12:03.947 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:02.648+0000 7f658d7fa700 1 -- 172.21.15.29:0/771528107 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6584010680 con 0x7f6590102730 2024-03-18T01:12:03.947 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:02.648+0000 7f658d7fa700 1 -- 172.21.15.29:0/771528107 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6584010cb0 con 0x7f6590102730 2024-03-18T01:12:03.947 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:02.648+0000 7f6596f6e700 1 -- 172.21.15.29:0/771528107 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6590077180 con 0x7f6590102730 2024-03-18T01:12:03.947 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:02.648+0000 7f658d7fa700 1 -- 172.21.15.29:0/771528107 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f658401ad40 con 0x7f6590102730 2024-03-18T01:12:03.947 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:02.648+0000 7f6596f6e700 1 -- 172.21.15.29:0/771528107 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6590077670 con 0x7f6590102730 2024-03-18T01:12:03.948 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:02.648+0000 7f658d7fa700 1 -- 172.21.15.29:0/771528107 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 12) v1 ==== 43381+0+0 (secure 0 0 0) 0x7f6584025400 con 0x7f6590102730 2024-03-18T01:12:03.948 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:02.649+0000 7f6596f6e700 1 -- 172.21.15.29:0/771528107 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f659004bed0 con 0x7f6590102730 2024-03-18T01:12:03.948 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:02.649+0000 7f658d7fa700 1 --2- 172.21.15.29:0/771528107 >> [v2:172.21.15.29:6800/2687730237,v1:172.21.15.29:6801/2687730237] conn(0x7f65800367e0 0x7f6580038c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:12:03.948 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:02.649+0000 7f658d7fa700 1 -- 172.21.15.29:0/771528107 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f658404a3c0 con 0x7f6590102730 2024-03-18T01:12:03.948 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:02.649+0000 7f658ffff700 1 --2- 172.21.15.29:0/771528107 >> [v2:172.21.15.29:6800/2687730237,v1:172.21.15.29:6801/2687730237] conn(0x7f65800367e0 0x7f6580038c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:12:03.948 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:02.649+0000 7f658ffff700 1 --2- 172.21.15.29:0/771528107 >> [v2:172.21.15.29:6800/2687730237,v1:172.21.15.29:6801/2687730237] conn(0x7f65800367e0 0x7f6580038c90 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f657c009990 tx=0x7f657c006e40 comp rx=0 tx=0).ready entity=mgr.14158 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:12:03.949 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:02.653+0000 7f658d7fa700 1 -- 172.21.15.29:0/771528107 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175520 (secure 0 0 0) 0x7f658402c8f0 con 0x7f6590102730 2024-03-18T01:12:03.949 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:02.864+0000 7f6596f6e700 1 -- 172.21.15.29:0/771528107 --> [v2:172.21.15.29:6800/2687730237,v1:172.21.15.29:6801/2687730237] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f659005f410 con 0x7f65800367e0 2024-03-18T01:12:03.949 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:03.110+0000 7f658d7fa700 1 -- 172.21.15.29:0/771528107 <== mgr.14158 v2:172.21.15.29:6800/2687730237 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+252 (secure 0 0 0) 0x7f659005f410 con 0x7f65800367e0 2024-03-18T01:12:03.949 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:03.112+0000 7f6596f6e700 1 -- 172.21.15.29:0/771528107 >> [v2:172.21.15.29:6800/2687730237,v1:172.21.15.29:6801/2687730237] conn(0x7f65800367e0 msgr2=0x7f6580038c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:12:03.949 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:03.112+0000 7f6596f6e700 1 --2- 172.21.15.29:0/771528107 >> [v2:172.21.15.29:6800/2687730237,v1:172.21.15.29:6801/2687730237] conn(0x7f65800367e0 0x7f6580038c90 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f657c009990 tx=0x7f657c006e40 comp rx=0 tx=0).stop 2024-03-18T01:12:03.949 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:03.112+0000 7f6596f6e700 1 -- 172.21.15.29:0/771528107 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6590102730 msgr2=0x7f6590078bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:12:03.949 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:03.112+0000 7f6596f6e700 1 --2- 172.21.15.29:0/771528107 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6590102730 0x7f6590078bd0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f65840094c0 tx=0x7f6584009250 comp rx=0 tx=0).stop 2024-03-18T01:12:03.950 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:03.112+0000 7f6596f6e700 1 -- 172.21.15.29:0/771528107 shutdown_connections 2024-03-18T01:12:03.950 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:03.112+0000 7f6596f6e700 1 --2- 172.21.15.29:0/771528107 >> [v2:172.21.15.29:6800/2687730237,v1:172.21.15.29:6801/2687730237] conn(0x7f65800367e0 0x7f6580038c90 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:12:03.950 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:03.112+0000 7f6596f6e700 1 --2- 172.21.15.29:0/771528107 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6590102730 0x7f6590078bd0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:12:03.950 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:03.112+0000 7f6596f6e700 1 -- 172.21.15.29:0/771528107 >> 172.21.15.29:0/771528107 conn(0x7f65900fdf10 msgr2=0x7f65900feab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:12:03.950 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:03.112+0000 7f6596f6e700 1 -- 172.21.15.29:0/771528107 shutdown_connections 2024-03-18T01:12:03.951 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:03.112+0000 7f6596f6e700 1 -- 172.21.15.29:0/771528107 wait complete. 2024-03-18T01:12:03.951 INFO:teuthology.orchestra.run.smithi029.stdout:Fetching dashboard port number... 2024-03-18T01:12:04.181 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:12:04 smithi029 ceph-mon[29331]: from='client.14172 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T01:12:04.181 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:12:04 smithi029 ceph-mon[29331]: from='mgr.14158 172.21.15.29:0/4059439045' entity='mgr.smithi029.qviwhl' 2024-03-18T01:12:05.467 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:12:05 smithi029 ceph-mon[29331]: from='client.? 172.21.15.29:0/2940525708' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-03-18T01:12:05.475 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout 8443 2024-03-18T01:12:05.475 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:04.392+0000 7f6e44da5700 1 Processor -- start 2024-03-18T01:12:05.475 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:04.392+0000 7f6e44da5700 1 -- start start 2024-03-18T01:12:05.475 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:04.393+0000 7f6e44da5700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6e40102740 0x7f6e40102b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:12:05.476 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:04.393+0000 7f6e44da5700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e40103090 con 0x7f6e40102740 2024-03-18T01:12:05.476 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:04.393+0000 7f6e3e59c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6e40102740 0x7f6e40102b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:12:05.476 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:04.393+0000 7f6e3e59c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6e40102740 0x7f6e40102b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:55468/0 (socket says 172.21.15.29:55468) 2024-03-18T01:12:05.476 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:04.393+0000 7f6e3e59c700 1 -- 172.21.15.29:0/1393835836 learned_addr learned my addr 172.21.15.29:0/1393835836 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-18T01:12:05.476 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:04.393+0000 7f6e3e59c700 1 -- 172.21.15.29:0/1393835836 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e401031d0 con 0x7f6e40102740 2024-03-18T01:12:05.476 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:04.393+0000 7f6e3e59c700 1 --2- 172.21.15.29:0/1393835836 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6e40102740 0x7f6e40102b50 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f6e300098e0 tx=0x7f6e30009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2612c1509a58bd1f server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:12:05.476 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:04.393+0000 7f6e3d59a700 1 -- 172.21.15.29:0/1393835836 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6e3000b860 con 0x7f6e40102740 2024-03-18T01:12:05.476 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:04.393+0000 7f6e3d59a700 1 -- 172.21.15.29:0/1393835836 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6e3000be90 con 0x7f6e40102740 2024-03-18T01:12:05.476 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:04.393+0000 7f6e3d59a700 1 -- 172.21.15.29:0/1393835836 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6e300103c0 con 0x7f6e40102740 2024-03-18T01:12:05.476 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:04.394+0000 7f6e44da5700 1 -- 172.21.15.29:0/1393835836 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6e40102740 msgr2=0x7f6e40102b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:12:05.476 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:04.394+0000 7f6e44da5700 1 --2- 172.21.15.29:0/1393835836 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6e40102740 0x7f6e40102b50 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f6e300098e0 tx=0x7f6e30009250 comp rx=0 tx=0).stop 2024-03-18T01:12:05.476 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:04.394+0000 7f6e44da5700 1 -- 172.21.15.29:0/1393835836 shutdown_connections 2024-03-18T01:12:05.477 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:04.394+0000 7f6e44da5700 1 --2- 172.21.15.29:0/1393835836 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6e40102740 0x7f6e40102b50 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:12:05.477 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:04.394+0000 7f6e44da5700 1 -- 172.21.15.29:0/1393835836 >> 172.21.15.29:0/1393835836 conn(0x7f6e400fdea0 msgr2=0x7f6e401002d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:12:05.477 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:04.394+0000 7f6e44da5700 1 -- 172.21.15.29:0/1393835836 shutdown_connections 2024-03-18T01:12:05.477 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:04.394+0000 7f6e44da5700 1 -- 172.21.15.29:0/1393835836 wait complete. 2024-03-18T01:12:05.477 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:04.395+0000 7f6e44da5700 1 Processor -- start 2024-03-18T01:12:05.477 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:04.395+0000 7f6e44da5700 1 -- start start 2024-03-18T01:12:05.477 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:04.395+0000 7f6e44da5700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6e40102740 0x7f6e401115e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:12:05.477 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:04.395+0000 7f6e44da5700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e40111b20 con 0x7f6e40102740 2024-03-18T01:12:05.477 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:04.395+0000 7f6e3e59c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6e40102740 0x7f6e401115e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:12:05.477 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:04.395+0000 7f6e3e59c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6e40102740 0x7f6e401115e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:55476/0 (socket says 172.21.15.29:55476) 2024-03-18T01:12:05.477 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:04.395+0000 7f6e3e59c700 1 -- 172.21.15.29:0/2940525708 learned_addr learned my addr 172.21.15.29:0/2940525708 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-18T01:12:05.477 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:04.396+0000 7f6e3e59c700 1 -- 172.21.15.29:0/2940525708 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e300094f0 con 0x7f6e40102740 2024-03-18T01:12:05.477 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:04.396+0000 7f6e3e59c700 1 --2- 172.21.15.29:0/2940525708 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6e40102740 0x7f6e401115e0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f6e30011c10 tx=0x7f6e30009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:12:05.477 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:04.396+0000 7f6e2effd700 1 -- 172.21.15.29:0/2940525708 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6e30010890 con 0x7f6e40102740 2024-03-18T01:12:05.477 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:04.396+0000 7f6e2effd700 1 -- 172.21.15.29:0/2940525708 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6e30010ec0 con 0x7f6e40102740 2024-03-18T01:12:05.478 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:04.396+0000 7f6e44da5700 1 -- 172.21.15.29:0/2940525708 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e40111d20 con 0x7f6e40102740 2024-03-18T01:12:05.478 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:04.396+0000 7f6e2effd700 1 -- 172.21.15.29:0/2940525708 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6e300243f0 con 0x7f6e40102740 2024-03-18T01:12:05.478 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:04.396+0000 7f6e44da5700 1 -- 172.21.15.29:0/2940525708 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e40112890 con 0x7f6e40102740 2024-03-18T01:12:05.478 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:04.397+0000 7f6e2effd700 1 -- 172.21.15.29:0/2940525708 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 12) v1 ==== 43381+0+0 (secure 0 0 0) 0x7f6e300109f0 con 0x7f6e40102740 2024-03-18T01:12:05.478 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:04.397+0000 7f6e44da5700 1 -- 172.21.15.29:0/2940525708 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e4005f5f0 con 0x7f6e40102740 2024-03-18T01:12:05.478 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:04.397+0000 7f6e2effd700 1 --2- 172.21.15.29:0/2940525708 >> [v2:172.21.15.29:6800/2687730237,v1:172.21.15.29:6801/2687730237] conn(0x7f6e280367e0 0x7f6e28038c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:12:05.478 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:04.397+0000 7f6e2effd700 1 -- 172.21.15.29:0/2940525708 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f6e3004a3a0 con 0x7f6e40102740 2024-03-18T01:12:05.478 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:04.397+0000 7f6e3dd9b700 1 --2- 172.21.15.29:0/2940525708 >> [v2:172.21.15.29:6800/2687730237,v1:172.21.15.29:6801/2687730237] conn(0x7f6e280367e0 0x7f6e28038c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:12:05.478 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:04.397+0000 7f6e3dd9b700 1 --2- 172.21.15.29:0/2940525708 >> [v2:172.21.15.29:6800/2687730237,v1:172.21.15.29:6801/2687730237] conn(0x7f6e280367e0 0x7f6e28038c90 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f6e34009990 tx=0x7f6e34006e40 comp rx=0 tx=0).ready entity=mgr.14158 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:12:05.478 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:04.402+0000 7f6e2effd700 1 -- 172.21.15.29:0/2940525708 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175520 (secure 0 0 0) 0x7f6e3002b940 con 0x7f6e40102740 2024-03-18T01:12:05.478 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:04.595+0000 7f6e44da5700 1 -- 172.21.15.29:0/2940525708 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) v1 -- 0x7f6e4004bed0 con 0x7f6e40102740 2024-03-18T01:12:05.478 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:04.595+0000 7f6e2effd700 1 -- 172.21.15.29:0/2940525708 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) v1 ==== 112+0+5 (secure 0 0 0) 0x7f6e3002b2f0 con 0x7f6e40102740 2024-03-18T01:12:05.478 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:04.597+0000 7f6e44da5700 1 -- 172.21.15.29:0/2940525708 >> [v2:172.21.15.29:6800/2687730237,v1:172.21.15.29:6801/2687730237] conn(0x7f6e280367e0 msgr2=0x7f6e28038c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:12:05.479 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:04.597+0000 7f6e44da5700 1 --2- 172.21.15.29:0/2940525708 >> [v2:172.21.15.29:6800/2687730237,v1:172.21.15.29:6801/2687730237] conn(0x7f6e280367e0 0x7f6e28038c90 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f6e34009990 tx=0x7f6e34006e40 comp rx=0 tx=0).stop 2024-03-18T01:12:05.479 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:04.597+0000 7f6e44da5700 1 -- 172.21.15.29:0/2940525708 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6e40102740 msgr2=0x7f6e401115e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:12:05.479 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:04.597+0000 7f6e44da5700 1 --2- 172.21.15.29:0/2940525708 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6e40102740 0x7f6e401115e0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f6e30011c10 tx=0x7f6e30009250 comp rx=0 tx=0).stop 2024-03-18T01:12:05.479 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:04.597+0000 7f6e44da5700 1 -- 172.21.15.29:0/2940525708 shutdown_connections 2024-03-18T01:12:05.479 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:04.597+0000 7f6e44da5700 1 --2- 172.21.15.29:0/2940525708 >> [v2:172.21.15.29:6800/2687730237,v1:172.21.15.29:6801/2687730237] conn(0x7f6e280367e0 0x7f6e28038c90 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:12:05.479 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:04.597+0000 7f6e44da5700 1 --2- 172.21.15.29:0/2940525708 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6e40102740 0x7f6e401115e0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:12:05.479 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:04.597+0000 7f6e44da5700 1 -- 172.21.15.29:0/2940525708 >> 172.21.15.29:0/2940525708 conn(0x7f6e400fdea0 msgr2=0x7f6e400fea30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:12:05.479 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:04.598+0000 7f6e44da5700 1 -- 172.21.15.29:0/2940525708 shutdown_connections 2024-03-18T01:12:05.479 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:04.598+0000 7f6e44da5700 1 -- 172.21.15.29:0/2940525708 wait complete. 2024-03-18T01:12:05.486 INFO:teuthology.orchestra.run.smithi029.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-18T01:12:05.486 INFO:teuthology.orchestra.run.smithi029.stdout:systemctl: stdout disabled 2024-03-18T01:12:05.494 INFO:teuthology.orchestra.run.smithi029.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-18T01:12:05.494 INFO:teuthology.orchestra.run.smithi029.stdout:systemctl: stdout inactive 2024-03-18T01:12:05.494 INFO:teuthology.orchestra.run.smithi029.stdout:firewalld.service is not enabled 2024-03-18T01:12:05.495 INFO:teuthology.orchestra.run.smithi029.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-03-18T01:12:05.497 INFO:teuthology.orchestra.run.smithi029.stdout:Ceph Dashboard is now available at: 2024-03-18T01:12:05.497 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-18T01:12:05.497 INFO:teuthology.orchestra.run.smithi029.stdout: URL: https://smithi029.front.sepia.ceph.com:8443/ 2024-03-18T01:12:05.497 INFO:teuthology.orchestra.run.smithi029.stdout: User: admin 2024-03-18T01:12:05.497 INFO:teuthology.orchestra.run.smithi029.stdout: Password: s747vxnrs7 2024-03-18T01:12:05.497 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-18T01:12:05.497 INFO:teuthology.orchestra.run.smithi029.stdout:Saving cluster configuration to /var/lib/ceph/37f51182-e4c4-11ee-95c9-87774f69a715/config directory 2024-03-18T01:12:05.498 INFO:teuthology.orchestra.run.smithi029.stdout:Enabling autotune for osd_memory_target 2024-03-18T01:12:06.714 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:05.915+0000 7f6c836b5700 1 Processor -- start 2024-03-18T01:12:06.714 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:05.915+0000 7f6c836b5700 1 -- start start 2024-03-18T01:12:06.714 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:05.915+0000 7f6c836b5700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6c7c076bf0 0x7f6c7c1076e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:12:06.714 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:05.915+0000 7f6c836b5700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c7c1030d0 con 0x7f6c7c076bf0 2024-03-18T01:12:06.714 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:05.915+0000 7f6c81451700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6c7c076bf0 0x7f6c7c1076e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:12:06.715 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:05.915+0000 7f6c81451700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6c7c076bf0 0x7f6c7c1076e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:55486/0 (socket says 172.21.15.29:55486) 2024-03-18T01:12:06.715 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:05.915+0000 7f6c81451700 1 -- 172.21.15.29:0/2371443834 learned_addr learned my addr 172.21.15.29:0/2371443834 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-18T01:12:06.715 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:05.915+0000 7f6c81451700 1 -- 172.21.15.29:0/2371443834 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c7c107c20 con 0x7f6c7c076bf0 2024-03-18T01:12:06.715 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:05.916+0000 7f6c81451700 1 --2- 172.21.15.29:0/2371443834 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6c7c076bf0 0x7f6c7c1076e0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f6c780098e0 tx=0x7f6c78009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45273a75904253ff server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:12:06.715 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:05.916+0000 7f6c73fff700 1 -- 172.21.15.29:0/2371443834 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6c7800b860 con 0x7f6c7c076bf0 2024-03-18T01:12:06.715 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:05.916+0000 7f6c73fff700 1 -- 172.21.15.29:0/2371443834 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6c7800be90 con 0x7f6c7c076bf0 2024-03-18T01:12:06.715 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:05.916+0000 7f6c73fff700 1 -- 172.21.15.29:0/2371443834 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6c780103c0 con 0x7f6c7c076bf0 2024-03-18T01:12:06.715 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:05.916+0000 7f6c836b5700 1 -- 172.21.15.29:0/2371443834 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6c7c076bf0 msgr2=0x7f6c7c1076e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:12:06.715 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:05.916+0000 7f6c836b5700 1 --2- 172.21.15.29:0/2371443834 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6c7c076bf0 0x7f6c7c1076e0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f6c780098e0 tx=0x7f6c78009250 comp rx=0 tx=0).stop 2024-03-18T01:12:06.715 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:05.916+0000 7f6c836b5700 1 -- 172.21.15.29:0/2371443834 shutdown_connections 2024-03-18T01:12:06.715 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:05.916+0000 7f6c836b5700 1 --2- 172.21.15.29:0/2371443834 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6c7c076bf0 0x7f6c7c1076e0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:12:06.715 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:05.916+0000 7f6c836b5700 1 -- 172.21.15.29:0/2371443834 >> 172.21.15.29:0/2371443834 conn(0x7f6c7c0fcd70 msgr2=0x7f6c7c0ff180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:12:06.715 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:05.917+0000 7f6c836b5700 1 -- 172.21.15.29:0/2371443834 shutdown_connections 2024-03-18T01:12:06.715 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:05.917+0000 7f6c836b5700 1 -- 172.21.15.29:0/2371443834 wait complete. 2024-03-18T01:12:06.716 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:05.917+0000 7f6c836b5700 1 Processor -- start 2024-03-18T01:12:06.716 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:05.917+0000 7f6c836b5700 1 -- start start 2024-03-18T01:12:06.716 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:05.917+0000 7f6c836b5700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6c7c076bf0 0x7f6c7c1137d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:12:06.716 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:05.917+0000 7f6c836b5700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c7c113d10 con 0x7f6c7c076bf0 2024-03-18T01:12:06.716 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:05.917+0000 7f6c81451700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6c7c076bf0 0x7f6c7c1137d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:12:06.716 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:05.917+0000 7f6c81451700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6c7c076bf0 0x7f6c7c1137d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:55492/0 (socket says 172.21.15.29:55492) 2024-03-18T01:12:06.716 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:05.917+0000 7f6c81451700 1 -- 172.21.15.29:0/1416397530 learned_addr learned my addr 172.21.15.29:0/1416397530 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-18T01:12:06.716 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:05.918+0000 7f6c81451700 1 -- 172.21.15.29:0/1416397530 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c780094f0 con 0x7f6c7c076bf0 2024-03-18T01:12:06.716 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:05.918+0000 7f6c81451700 1 --2- 172.21.15.29:0/1416397530 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6c7c076bf0 0x7f6c7c1137d0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f6c780094c0 tx=0x7f6c78009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:12:06.716 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:05.918+0000 7f6c71ffb700 1 -- 172.21.15.29:0/1416397530 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6c78010890 con 0x7f6c7c076bf0 2024-03-18T01:12:06.716 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:05.918+0000 7f6c71ffb700 1 -- 172.21.15.29:0/1416397530 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6c78010ec0 con 0x7f6c7c076bf0 2024-03-18T01:12:06.716 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:05.918+0000 7f6c836b5700 1 -- 172.21.15.29:0/1416397530 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c7c113f10 con 0x7f6c7c076bf0 2024-03-18T01:12:06.716 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:05.918+0000 7f6c71ffb700 1 -- 172.21.15.29:0/1416397530 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6c780243f0 con 0x7f6c7c076bf0 2024-03-18T01:12:06.716 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:05.918+0000 7f6c836b5700 1 -- 172.21.15.29:0/1416397530 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c7c114a80 con 0x7f6c7c076bf0 2024-03-18T01:12:06.716 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:05.919+0000 7f6c71ffb700 1 -- 172.21.15.29:0/1416397530 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 12) v1 ==== 43381+0+0 (secure 0 0 0) 0x7f6c7802d3f0 con 0x7f6c7c076bf0 2024-03-18T01:12:06.716 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:05.919+0000 7f6c71ffb700 1 --2- 172.21.15.29:0/1416397530 >> [v2:172.21.15.29:6800/2687730237,v1:172.21.15.29:6801/2687730237] conn(0x7f6c68036830 0x7f6c68038ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:12:06.717 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:05.919+0000 7f6c71ffb700 1 -- 172.21.15.29:0/1416397530 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f6c7801a7e0 con 0x7f6c7c076bf0 2024-03-18T01:12:06.717 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:05.919+0000 7f6c80c50700 1 --2- 172.21.15.29:0/1416397530 >> [v2:172.21.15.29:6800/2687730237,v1:172.21.15.29:6801/2687730237] conn(0x7f6c68036830 0x7f6c68038ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:12:06.717 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:05.919+0000 7f6c836b5700 1 -- 172.21.15.29:0/1416397530 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c7c05f5f0 con 0x7f6c7c076bf0 2024-03-18T01:12:06.717 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:05.920+0000 7f6c80c50700 1 --2- 172.21.15.29:0/1416397530 >> [v2:172.21.15.29:6800/2687730237,v1:172.21.15.29:6801/2687730237] conn(0x7f6c68036830 0x7f6c68038ce0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f6c6c009990 tx=0x7f6c6c006e40 comp rx=0 tx=0).ready entity=mgr.14158 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:12:06.717 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:05.924+0000 7f6c71ffb700 1 -- 172.21.15.29:0/1416397530 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175520 (secure 0 0 0) 0x7f6c7802b8e0 con 0x7f6c7c076bf0 2024-03-18T01:12:06.717 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:06.112+0000 7f6c71ffb700 1 -- 172.21.15.29:0/1416397530 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mgrmap(e 13) v1 ==== 43424+0+0 (secure 0 0 0) 0x7f6c7802d740 con 0x7f6c7c076bf0 2024-03-18T01:12:06.717 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:06.118+0000 7f6c836b5700 1 -- 172.21.15.29:0/1416397530 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command([{prefix=config set, name=osd_memory_target_autotune}] v 0) v1 -- 0x7f6c7c04bed0 con 0x7f6c7c076bf0 2024-03-18T01:12:06.717 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:06.118+0000 7f6c71ffb700 1 -- 172.21.15.29:0/1416397530 <== mon.0 v2:172.21.15.29:3300/0 8 ==== mon_command_ack([{prefix=config set, name=osd_memory_target_autotune}]=0 v8)=0 v8) v1 ==== 127+0+0 (secure 0 0 0) 0x7f6c7802b8e0 con 0x7f6c7c076bf0 2024-03-18T01:12:06.717 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:06.120+0000 7f6c836b5700 1 -- 172.21.15.29:0/1416397530 >> [v2:172.21.15.29:6800/2687730237,v1:172.21.15.29:6801/2687730237] conn(0x7f6c68036830 msgr2=0x7f6c68038ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:12:06.717 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:06.120+0000 7f6c836b5700 1 --2- 172.21.15.29:0/1416397530 >> [v2:172.21.15.29:6800/2687730237,v1:172.21.15.29:6801/2687730237] conn(0x7f6c68036830 0x7f6c68038ce0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f6c6c009990 tx=0x7f6c6c006e40 comp rx=0 tx=0).stop 2024-03-18T01:12:06.717 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:06.120+0000 7f6c836b5700 1 -- 172.21.15.29:0/1416397530 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6c7c076bf0 msgr2=0x7f6c7c1137d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:12:06.717 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:06.120+0000 7f6c836b5700 1 --2- 172.21.15.29:0/1416397530 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6c7c076bf0 0x7f6c7c1137d0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f6c780094c0 tx=0x7f6c78009250 comp rx=0 tx=0).stop 2024-03-18T01:12:06.717 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:06.120+0000 7f6c836b5700 1 -- 172.21.15.29:0/1416397530 shutdown_connections 2024-03-18T01:12:06.717 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:06.120+0000 7f6c836b5700 1 --2- 172.21.15.29:0/1416397530 >> [v2:172.21.15.29:6800/2687730237,v1:172.21.15.29:6801/2687730237] conn(0x7f6c68036830 0x7f6c68038ce0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:12:06.717 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:06.120+0000 7f6c836b5700 1 --2- 172.21.15.29:0/1416397530 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6c7c076bf0 0x7f6c7c1137d0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:12:06.718 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:06.120+0000 7f6c836b5700 1 -- 172.21.15.29:0/1416397530 >> 172.21.15.29:0/1416397530 conn(0x7f6c7c0fcd70 msgr2=0x7f6c7c0fd970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:12:06.718 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:06.121+0000 7f6c836b5700 1 -- 172.21.15.29:0/1416397530 shutdown_connections 2024-03-18T01:12:06.718 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:06.121+0000 7f6c836b5700 1 -- 172.21.15.29:0/1416397530 wait complete. 2024-03-18T01:12:07.318 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:12:07 smithi029 ceph-mon[29331]: mgrmap e13: smithi029.qviwhl(active, since 6s) 2024-03-18T01:12:08.054 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:07.128+0000 7f84d14cc700 1 Processor -- start 2024-03-18T01:12:08.055 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:07.128+0000 7f84d14cc700 1 -- start start 2024-03-18T01:12:08.055 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:07.129+0000 7f84d14cc700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f84cc103cd0 0x7f84cc1040e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:12:08.055 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:07.129+0000 7f84d14cc700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84cc104620 con 0x7f84cc103cd0 2024-03-18T01:12:08.055 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:07.129+0000 7f84caffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f84cc103cd0 0x7f84cc1040e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:12:08.055 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:07.129+0000 7f84caffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f84cc103cd0 0x7f84cc1040e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:55504/0 (socket says 172.21.15.29:55504) 2024-03-18T01:12:08.055 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:07.129+0000 7f84caffd700 1 -- 172.21.15.29:0/2484889616 learned_addr learned my addr 172.21.15.29:0/2484889616 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-18T01:12:08.055 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:07.129+0000 7f84caffd700 1 -- 172.21.15.29:0/2484889616 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84cc104760 con 0x7f84cc103cd0 2024-03-18T01:12:08.055 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:07.129+0000 7f84caffd700 1 --2- 172.21.15.29:0/2484889616 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f84cc103cd0 0x7f84cc1040e0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f84bc0098e0 tx=0x7f84bc009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c58df182bda6550 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:12:08.055 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:07.129+0000 7f84c9ffb700 1 -- 172.21.15.29:0/2484889616 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f84bc00b860 con 0x7f84cc103cd0 2024-03-18T01:12:08.055 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:07.129+0000 7f84c9ffb700 1 -- 172.21.15.29:0/2484889616 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f84bc00be90 con 0x7f84cc103cd0 2024-03-18T01:12:08.055 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:07.129+0000 7f84c9ffb700 1 -- 172.21.15.29:0/2484889616 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f84bc0103c0 con 0x7f84cc103cd0 2024-03-18T01:12:08.055 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:07.130+0000 7f84d14cc700 1 -- 172.21.15.29:0/2484889616 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f84cc103cd0 msgr2=0x7f84cc1040e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:12:08.055 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:07.130+0000 7f84d14cc700 1 --2- 172.21.15.29:0/2484889616 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f84cc103cd0 0x7f84cc1040e0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f84bc0098e0 tx=0x7f84bc009250 comp rx=0 tx=0).stop 2024-03-18T01:12:08.056 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:07.130+0000 7f84d14cc700 1 -- 172.21.15.29:0/2484889616 shutdown_connections 2024-03-18T01:12:08.056 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:07.130+0000 7f84d14cc700 1 --2- 172.21.15.29:0/2484889616 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f84cc103cd0 0x7f84cc1040e0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:12:08.056 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:07.130+0000 7f84d14cc700 1 -- 172.21.15.29:0/2484889616 >> 172.21.15.29:0/2484889616 conn(0x7f84cc0fc4d0 msgr2=0x7f84cc0fe8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:12:08.056 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:07.130+0000 7f84d14cc700 1 -- 172.21.15.29:0/2484889616 shutdown_connections 2024-03-18T01:12:08.056 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:07.130+0000 7f84d14cc700 1 -- 172.21.15.29:0/2484889616 wait complete. 2024-03-18T01:12:08.056 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:07.131+0000 7f84d14cc700 1 Processor -- start 2024-03-18T01:12:08.056 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:07.131+0000 7f84d14cc700 1 -- start start 2024-03-18T01:12:08.056 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:07.131+0000 7f84d14cc700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f84cc103cd0 0x7f84cc1033e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:12:08.056 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:07.131+0000 7f84d14cc700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84cc103920 con 0x7f84cc103cd0 2024-03-18T01:12:08.056 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:07.131+0000 7f84caffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f84cc103cd0 0x7f84cc1033e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:12:08.056 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:07.131+0000 7f84caffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f84cc103cd0 0x7f84cc1033e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:55520/0 (socket says 172.21.15.29:55520) 2024-03-18T01:12:08.056 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:07.131+0000 7f84caffd700 1 -- 172.21.15.29:0/3495105108 learned_addr learned my addr 172.21.15.29:0/3495105108 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-18T01:12:08.056 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:07.131+0000 7f84caffd700 1 -- 172.21.15.29:0/3495105108 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84bc0094f0 con 0x7f84cc103cd0 2024-03-18T01:12:08.056 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:07.132+0000 7f84caffd700 1 --2- 172.21.15.29:0/3495105108 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f84cc103cd0 0x7f84cc1033e0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f84bc0090b0 tx=0x7f84bc009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:12:08.056 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:07.132+0000 7f84b37fe700 1 -- 172.21.15.29:0/3495105108 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f84bc010840 con 0x7f84cc103cd0 2024-03-18T01:12:08.057 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:07.132+0000 7f84b37fe700 1 -- 172.21.15.29:0/3495105108 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f84bc010e70 con 0x7f84cc103cd0 2024-03-18T01:12:08.057 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:07.132+0000 7f84d14cc700 1 -- 172.21.15.29:0/3495105108 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84cc101990 con 0x7f84cc103cd0 2024-03-18T01:12:08.057 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:07.132+0000 7f84b37fe700 1 -- 172.21.15.29:0/3495105108 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f84bc024e40 con 0x7f84cc103cd0 2024-03-18T01:12:08.057 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:07.132+0000 7f84d14cc700 1 -- 172.21.15.29:0/3495105108 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84cc101e80 con 0x7f84cc103cd0 2024-03-18T01:12:08.057 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:07.133+0000 7f84b37fe700 1 -- 172.21.15.29:0/3495105108 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 13) v1 ==== 43424+0+0 (secure 0 0 0) 0x7f84bc01a520 con 0x7f84cc103cd0 2024-03-18T01:12:08.057 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:07.133+0000 7f84b37fe700 1 --2- 172.21.15.29:0/3495105108 >> [v2:172.21.15.29:6800/2687730237,v1:172.21.15.29:6801/2687730237] conn(0x7f84b4036880 0x7f84b4038d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:12:08.057 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:07.133+0000 7f84b37fe700 1 -- 172.21.15.29:0/3495105108 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f84bc04aa90 con 0x7f84cc103cd0 2024-03-18T01:12:08.057 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:07.133+0000 7f84d14cc700 1 -- 172.21.15.29:0/3495105108 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84cc04bed0 con 0x7f84cc103cd0 2024-03-18T01:12:08.057 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:07.133+0000 7f84ca7fc700 1 --2- 172.21.15.29:0/3495105108 >> [v2:172.21.15.29:6800/2687730237,v1:172.21.15.29:6801/2687730237] conn(0x7f84b4036880 0x7f84b4038d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:12:08.057 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:07.133+0000 7f84ca7fc700 1 --2- 172.21.15.29:0/3495105108 >> [v2:172.21.15.29:6800/2687730237,v1:172.21.15.29:6801/2687730237] conn(0x7f84b4036880 0x7f84b4038d30 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f84c0009990 tx=0x7f84c0006e40 comp rx=0 tx=0).ready entity=mgr.14158 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:12:08.057 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:07.138+0000 7f84b37fe700 1 -- 172.21.15.29:0/3495105108 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175520 (secure 0 0 0) 0x7f84bc018940 con 0x7f84cc103cd0 2024-03-18T01:12:08.057 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:07.442+0000 7f84d14cc700 1 -- 172.21.15.29:0/3495105108 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) v1 -- 0x7f84cc1011e0 con 0x7f84cc103cd0 2024-03-18T01:12:08.057 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:07.444+0000 7f84b37fe700 1 -- 172.21.15.29:0/3495105108 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v33)=0 set mgr/dashboard/cluster/status v33) v1 ==== 153+0+0 (secure 0 0 0) 0x7f84bc018350 con 0x7f84cc103cd0 2024-03-18T01:12:08.057 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:07.446+0000 7f84d14cc700 1 -- 172.21.15.29:0/3495105108 >> [v2:172.21.15.29:6800/2687730237,v1:172.21.15.29:6801/2687730237] conn(0x7f84b4036880 msgr2=0x7f84b4038d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:12:08.057 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:07.446+0000 7f84d14cc700 1 --2- 172.21.15.29:0/3495105108 >> [v2:172.21.15.29:6800/2687730237,v1:172.21.15.29:6801/2687730237] conn(0x7f84b4036880 0x7f84b4038d30 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f84c0009990 tx=0x7f84c0006e40 comp rx=0 tx=0).stop 2024-03-18T01:12:08.058 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:07.446+0000 7f84d14cc700 1 -- 172.21.15.29:0/3495105108 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f84cc103cd0 msgr2=0x7f84cc1033e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:12:08.058 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:07.446+0000 7f84d14cc700 1 --2- 172.21.15.29:0/3495105108 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f84cc103cd0 0x7f84cc1033e0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f84bc0090b0 tx=0x7f84bc009250 comp rx=0 tx=0).stop 2024-03-18T01:12:08.058 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:07.446+0000 7f84d14cc700 1 -- 172.21.15.29:0/3495105108 shutdown_connections 2024-03-18T01:12:08.058 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:07.446+0000 7f84d14cc700 1 --2- 172.21.15.29:0/3495105108 >> [v2:172.21.15.29:6800/2687730237,v1:172.21.15.29:6801/2687730237] conn(0x7f84b4036880 0x7f84b4038d30 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:12:08.058 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:07.446+0000 7f84d14cc700 1 --2- 172.21.15.29:0/3495105108 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f84cc103cd0 0x7f84cc1033e0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:12:08.058 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:07.446+0000 7f84d14cc700 1 -- 172.21.15.29:0/3495105108 >> 172.21.15.29:0/3495105108 conn(0x7f84cc0fc4d0 msgr2=0x7f84cc0fd030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:12:08.058 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:07.446+0000 7f84d14cc700 1 -- 172.21.15.29:0/3495105108 shutdown_connections 2024-03-18T01:12:08.058 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-18T01:12:07.446+0000 7f84d14cc700 1 -- 172.21.15.29:0/3495105108 wait complete. 2024-03-18T01:12:08.058 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-03-18T01:12:08.058 INFO:teuthology.orchestra.run.smithi029.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-03-18T01:12:08.058 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-18T01:12:08.058 INFO:teuthology.orchestra.run.smithi029.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid 37f51182-e4c4-11ee-95c9-87774f69a715 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-03-18T01:12:08.058 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-18T01:12:08.058 INFO:teuthology.orchestra.run.smithi029.stdout:Or, if you are only running a single cluster on this host: 2024-03-18T01:12:08.058 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-18T01:12:08.059 INFO:teuthology.orchestra.run.smithi029.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-03-18T01:12:08.059 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-18T01:12:08.059 INFO:teuthology.orchestra.run.smithi029.stdout:Please consider enabling telemetry to help improve Ceph: 2024-03-18T01:12:08.059 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-18T01:12:08.059 INFO:teuthology.orchestra.run.smithi029.stdout: ceph telemetry on 2024-03-18T01:12:08.059 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-18T01:12:08.059 INFO:teuthology.orchestra.run.smithi029.stdout:For more information see: 2024-03-18T01:12:08.059 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-18T01:12:08.059 INFO:teuthology.orchestra.run.smithi029.stdout: https://docs.ceph.com/docs/master/mgr/telemetry/ 2024-03-18T01:12:08.059 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-18T01:12:08.059 INFO:teuthology.orchestra.run.smithi029.stdout:Bootstrap complete. 2024-03-18T01:12:08.095 INFO:tasks.cephadm:Fetching config... 2024-03-18T01:12:08.095 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-03-18T01:12:08.095 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-03-18T01:12:08.111 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-03-18T01:12:08.111 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-03-18T01:12:08.111 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-03-18T01:12:08.166 INFO:tasks.cephadm:Fetching mon keyring... 2024-03-18T01:12:08.166 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-03-18T01:12:08.166 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd if=/var/lib/ceph/37f51182-e4c4-11ee-95c9-87774f69a715/mon.smithi029/keyring of=/dev/stdout 2024-03-18T01:12:08.233 INFO:tasks.cephadm:Fetching pub ssh key... 2024-03-18T01:12:08.234 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-03-18T01:12:08.234 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-03-18T01:12:08.289 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-03-18T01:12:08.289 DEBUG:teuthology.orchestra.run.smithi029:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCi0PmGToElTL5kISz6f8jjktVyRjOxEv+aZlHENCJFo31FIHSrj08Y4ku93j5VusR0iTCsWeXc1QOE3eRqODKFOxauT/fcZqfsFobLgmyZI1soBQUjYcYFGNAyaE23RmAjil2FrsLWz+jZ1/jToiInfg6S+qyKIJ4cQecBGbYB6OfA93Hi3N3W8zDACJum2o/b6IPOk2oVRVRZnuS/lRoAI42rOZtmBWE9FXz1GG6uWgRkYQXmrZG+2eqt5AIj2NF6q6XdaOaspMPo2JISWkJRExZw4sNCbpj/xNP7zAnOYDvRaSo/rK2BE+yRSVwFM/Gd/51X2eDtkVWIitcs+DsVhAVWt8FFaIQG4YHiwDdhcv3OH5mjtEHbkQAFPiPR1EVrM66kK749iySeXXZsjxr703xejYXqMZygQHtU7XbTsXEGvDNcqCv2eZkp6WAzB1qIyGq41e87RxgSIg9zUkga6OFKZq4oLUPokZ2eIpEXRXgBVTbFqdKnGfdcPd10ZR8= ceph-37f51182-e4c4-11ee-95c9-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-18T01:12:08.368 INFO:teuthology.orchestra.run.smithi029.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCi0PmGToElTL5kISz6f8jjktVyRjOxEv+aZlHENCJFo31FIHSrj08Y4ku93j5VusR0iTCsWeXc1QOE3eRqODKFOxauT/fcZqfsFobLgmyZI1soBQUjYcYFGNAyaE23RmAjil2FrsLWz+jZ1/jToiInfg6S+qyKIJ4cQecBGbYB6OfA93Hi3N3W8zDACJum2o/b6IPOk2oVRVRZnuS/lRoAI42rOZtmBWE9FXz1GG6uWgRkYQXmrZG+2eqt5AIj2NF6q6XdaOaspMPo2JISWkJRExZw4sNCbpj/xNP7zAnOYDvRaSo/rK2BE+yRSVwFM/Gd/51X2eDtkVWIitcs+DsVhAVWt8FFaIQG4YHiwDdhcv3OH5mjtEHbkQAFPiPR1EVrM66kK749iySeXXZsjxr703xejYXqMZygQHtU7XbTsXEGvDNcqCv2eZkp6WAzB1qIyGq41e87RxgSIg9zUkga6OFKZq4oLUPokZ2eIpEXRXgBVTbFqdKnGfdcPd10ZR8= ceph-37f51182-e4c4-11ee-95c9-87774f69a715 2024-03-18T01:12:08.392 DEBUG:teuthology.orchestra.run.smithi090:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCi0PmGToElTL5kISz6f8jjktVyRjOxEv+aZlHENCJFo31FIHSrj08Y4ku93j5VusR0iTCsWeXc1QOE3eRqODKFOxauT/fcZqfsFobLgmyZI1soBQUjYcYFGNAyaE23RmAjil2FrsLWz+jZ1/jToiInfg6S+qyKIJ4cQecBGbYB6OfA93Hi3N3W8zDACJum2o/b6IPOk2oVRVRZnuS/lRoAI42rOZtmBWE9FXz1GG6uWgRkYQXmrZG+2eqt5AIj2NF6q6XdaOaspMPo2JISWkJRExZw4sNCbpj/xNP7zAnOYDvRaSo/rK2BE+yRSVwFM/Gd/51X2eDtkVWIitcs+DsVhAVWt8FFaIQG4YHiwDdhcv3OH5mjtEHbkQAFPiPR1EVrM66kK749iySeXXZsjxr703xejYXqMZygQHtU7XbTsXEGvDNcqCv2eZkp6WAzB1qIyGq41e87RxgSIg9zUkga6OFKZq4oLUPokZ2eIpEXRXgBVTbFqdKnGfdcPd10ZR8= ceph-37f51182-e4c4-11ee-95c9-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-18T01:12:08.438 INFO:teuthology.orchestra.run.smithi090.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCi0PmGToElTL5kISz6f8jjktVyRjOxEv+aZlHENCJFo31FIHSrj08Y4ku93j5VusR0iTCsWeXc1QOE3eRqODKFOxauT/fcZqfsFobLgmyZI1soBQUjYcYFGNAyaE23RmAjil2FrsLWz+jZ1/jToiInfg6S+qyKIJ4cQecBGbYB6OfA93Hi3N3W8zDACJum2o/b6IPOk2oVRVRZnuS/lRoAI42rOZtmBWE9FXz1GG6uWgRkYQXmrZG+2eqt5AIj2NF6q6XdaOaspMPo2JISWkJRExZw4sNCbpj/xNP7zAnOYDvRaSo/rK2BE+yRSVwFM/Gd/51X2eDtkVWIitcs+DsVhAVWt8FFaIQG4YHiwDdhcv3OH5mjtEHbkQAFPiPR1EVrM66kK749iySeXXZsjxr703xejYXqMZygQHtU7XbTsXEGvDNcqCv2eZkp6WAzB1qIyGq41e87RxgSIg9zUkga6OFKZq4oLUPokZ2eIpEXRXgBVTbFqdKnGfdcPd10ZR8= ceph-37f51182-e4c4-11ee-95c9-87774f69a715 2024-03-18T01:12:08.453 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 37f51182-e4c4-11ee-95c9-87774f69a715 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-03-18T01:12:08.708 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/37f51182-e4c4-11ee-95c9-87774f69a715/mon.smithi029/config 2024-03-18T01:12:08.818 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:12:08 smithi029 ceph-mon[29331]: from='client.? 172.21.15.29:0/3495105108' entity='client.admin' 2024-03-18T01:12:11.338 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:12:11.336+0000 7fe4e2aff700 1 -- 172.21.15.29:0/2222366808 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe4dc0fd650 msgr2=0x7fe4dc0ffa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:12:11.339 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:12:11.336+0000 7fe4e2aff700 1 --2- 172.21.15.29:0/2222366808 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe4dc0fd650 0x7fe4dc0ffa30 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fe4d00048c0 tx=0x7fe4d0009e30 comp rx=0 tx=0).stop 2024-03-18T01:12:11.339 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:12:11.337+0000 7fe4e2aff700 1 -- 172.21.15.29:0/2222366808 shutdown_connections 2024-03-18T01:12:11.339 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:12:11.337+0000 7fe4e2aff700 1 --2- 172.21.15.29:0/2222366808 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe4dc0fd650 0x7fe4dc0ffa30 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:12:11.339 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:12:11.337+0000 7fe4e2aff700 1 -- 172.21.15.29:0/2222366808 >> 172.21.15.29:0/2222366808 conn(0x7fe4dc0f70d0 msgr2=0x7fe4dc0f9500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:12:11.339 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:12:11.337+0000 7fe4e2aff700 1 -- 172.21.15.29:0/2222366808 shutdown_connections 2024-03-18T01:12:11.339 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:12:11.337+0000 7fe4e2aff700 1 -- 172.21.15.29:0/2222366808 wait complete. 2024-03-18T01:12:11.339 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:12:11.337+0000 7fe4e2aff700 1 Processor -- start 2024-03-18T01:12:11.339 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:12:11.337+0000 7fe4e2aff700 1 -- start start 2024-03-18T01:12:11.339 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:12:11.337+0000 7fe4e2aff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe4dc0fd650 0x7fe4dc0fcfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:12:11.339 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:12:11.337+0000 7fe4e2aff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4dc0fd4f0 con 0x7fe4dc0fd650 2024-03-18T01:12:11.339 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:12:11.338+0000 7fe4e089b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe4dc0fd650 0x7fe4dc0fcfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:12:11.339 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:12:11.338+0000 7fe4e089b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe4dc0fd650 0x7fe4dc0fcfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:55540/0 (socket says 172.21.15.29:55540) 2024-03-18T01:12:11.340 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:12:11.338+0000 7fe4e089b700 1 -- 172.21.15.29:0/3185098151 learned_addr learned my addr 172.21.15.29:0/3185098151 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-18T01:12:11.340 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:12:11.338+0000 7fe4e089b700 1 -- 172.21.15.29:0/3185098151 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4d0005040 con 0x7fe4dc0fd650 2024-03-18T01:12:11.340 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:12:11.338+0000 7fe4e089b700 1 --2- 172.21.15.29:0/3185098151 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe4dc0fd650 0x7fe4dc0fcfb0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fe4d0009680 tx=0x7fe4d0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:12:11.340 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:12:11.338+0000 7fe4d97fa700 1 -- 172.21.15.29:0/3185098151 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe4d000bed0 con 0x7fe4dc0fd650 2024-03-18T01:12:11.340 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:12:11.338+0000 7fe4d97fa700 1 -- 172.21.15.29:0/3185098151 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe4d000b5c0 con 0x7fe4dc0fd650 2024-03-18T01:12:11.340 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:12:11.338+0000 7fe4d97fa700 1 -- 172.21.15.29:0/3185098151 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe4d001c5b0 con 0x7fe4dc0fd650 2024-03-18T01:12:11.340 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:12:11.338+0000 7fe4e2aff700 1 -- 172.21.15.29:0/3185098151 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4dc0fb5c0 con 0x7fe4dc0fd650 2024-03-18T01:12:11.341 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:12:11.338+0000 7fe4e2aff700 1 -- 172.21.15.29:0/3185098151 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4dc0fbab0 con 0x7fe4dc0fd650 2024-03-18T01:12:11.341 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:12:11.339+0000 7fe4d97fa700 1 -- 172.21.15.29:0/3185098151 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 13) v1 ==== 43424+0+0 (secure 0 0 0) 0x7fe4d0013420 con 0x7fe4dc0fd650 2024-03-18T01:12:11.341 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:12:11.339+0000 7fe4e2aff700 1 -- 172.21.15.29:0/3185098151 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4dc102f60 con 0x7fe4dc0fd650 2024-03-18T01:12:11.341 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:12:11.339+0000 7fe4d97fa700 1 --2- 172.21.15.29:0/3185098151 >> [v2:172.21.15.29:6800/2687730237,v1:172.21.15.29:6801/2687730237] conn(0x7fe4cc036880 0x7fe4cc038d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:12:11.341 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:12:11.340+0000 7fe4d97fa700 1 -- 172.21.15.29:0/3185098151 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fe4d0049950 con 0x7fe4dc0fd650 2024-03-18T01:12:11.341 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:12:11.340+0000 7fe4dbfff700 1 --2- 172.21.15.29:0/3185098151 >> [v2:172.21.15.29:6800/2687730237,v1:172.21.15.29:6801/2687730237] conn(0x7fe4cc036880 0x7fe4cc038d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:12:11.341 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:12:11.340+0000 7fe4dbfff700 1 --2- 172.21.15.29:0/3185098151 >> [v2:172.21.15.29:6800/2687730237,v1:172.21.15.29:6801/2687730237] conn(0x7fe4cc036880 0x7fe4cc038d30 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fe4c8009990 tx=0x7fe4c8006e40 comp rx=0 tx=0).ready entity=mgr.14158 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:12:11.344 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:12:11.344+0000 7fe4d97fa700 1 -- 172.21.15.29:0/3185098151 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175520 (secure 0 0 0) 0x7fe4d0013a80 con 0x7fe4dc0fd650 2024-03-18T01:12:11.540 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:12:11.540+0000 7fe4e2aff700 1 -- 172.21.15.29:0/3185098151 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) v1 -- 0x7fe4dc05f5f0 con 0x7fe4dc0fd650 2024-03-18T01:12:11.543 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:12:11.543+0000 7fe4d97fa700 1 -- 172.21.15.29:0/3185098151 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) v1 ==== 125+0+0 (secure 0 0 0) 0x7fe4d0011370 con 0x7fe4dc0fd650 2024-03-18T01:12:11.546 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:12:11.545+0000 7fe4e2aff700 1 -- 172.21.15.29:0/3185098151 >> [v2:172.21.15.29:6800/2687730237,v1:172.21.15.29:6801/2687730237] conn(0x7fe4cc036880 msgr2=0x7fe4cc038d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:12:11.546 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:12:11.545+0000 7fe4e2aff700 1 --2- 172.21.15.29:0/3185098151 >> [v2:172.21.15.29:6800/2687730237,v1:172.21.15.29:6801/2687730237] conn(0x7fe4cc036880 0x7fe4cc038d30 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fe4c8009990 tx=0x7fe4c8006e40 comp rx=0 tx=0).stop 2024-03-18T01:12:11.546 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:12:11.545+0000 7fe4e2aff700 1 -- 172.21.15.29:0/3185098151 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe4dc0fd650 msgr2=0x7fe4dc0fcfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:12:11.546 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:12:11.545+0000 7fe4e2aff700 1 --2- 172.21.15.29:0/3185098151 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe4dc0fd650 0x7fe4dc0fcfb0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fe4d0009680 tx=0x7fe4d0009e30 comp rx=0 tx=0).stop 2024-03-18T01:12:11.546 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:12:11.545+0000 7fe4e2aff700 1 -- 172.21.15.29:0/3185098151 shutdown_connections 2024-03-18T01:12:11.546 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:12:11.545+0000 7fe4e2aff700 1 --2- 172.21.15.29:0/3185098151 >> [v2:172.21.15.29:6800/2687730237,v1:172.21.15.29:6801/2687730237] conn(0x7fe4cc036880 0x7fe4cc038d30 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:12:11.546 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:12:11.545+0000 7fe4e2aff700 1 --2- 172.21.15.29:0/3185098151 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe4dc0fd650 0x7fe4dc0fcfb0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:12:11.546 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:12:11.545+0000 7fe4e2aff700 1 -- 172.21.15.29:0/3185098151 >> 172.21.15.29:0/3185098151 conn(0x7fe4dc0f70d0 msgr2=0x7fe4dc103b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:12:11.546 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:12:11.545+0000 7fe4e2aff700 1 -- 172.21.15.29:0/3185098151 shutdown_connections 2024-03-18T01:12:11.546 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:12:11.545+0000 7fe4e2aff700 1 -- 172.21.15.29:0/3185098151 wait complete. 2024-03-18T01:12:12.250 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-03-18T01:12:12.251 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 37f51182-e4c4-11ee-95c9-87774f69a715 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-03-18T01:12:12.690 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/37f51182-e4c4-11ee-95c9-87774f69a715/mon.smithi029/config 2024-03-18T01:12:12.813 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:12:12 smithi029 ceph-mon[29331]: from='client.? 172.21.15.29:0/3185098151' entity='client.admin' 2024-03-18T01:12:13.745 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:12:13 smithi029 ceph-mon[29331]: from='mgr.14158 172.21.15.29:0/4059439045' entity='mgr.smithi029.qviwhl' 2024-03-18T01:12:13.745 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:12:13 smithi029 ceph-mon[29331]: from='mgr.14158 172.21.15.29:0/4059439045' entity='mgr.smithi029.qviwhl' 2024-03-18T01:12:13.745 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:12:13 smithi029 ceph-mon[29331]: from='mgr.14158 172.21.15.29:0/4059439045' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "config rm", "who": "osd/host:smithi029", "name": "osd_memory_target"}]: dispatch 2024-03-18T01:12:13.746 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:12:13 smithi029 ceph-mon[29331]: from='mgr.14158 172.21.15.29:0/4059439045' entity='mgr.smithi029.qviwhl' 2024-03-18T01:12:13.746 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:12:13 smithi029 ceph-mon[29331]: from='mgr.14158 172.21.15.29:0/4059439045' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi029", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-03-18T01:12:13.746 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:12:13 smithi029 ceph-mon[29331]: from='mgr.14158 172.21.15.29:0/4059439045' entity='mgr.smithi029.qviwhl' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi029", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2024-03-18T01:12:13.746 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:12:13 smithi029 ceph-mon[29331]: from='mgr.14158 172.21.15.29:0/4059439045' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:12:13.746 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:12:13 smithi029 ceph-mon[29331]: Deploying daemon crash.smithi029 on smithi029 2024-03-18T01:12:15.447 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:12:15.444+0000 7fe42bf74700 1 -- 172.21.15.29:0/2973198699 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe4240fd040 msgr2=0x7fe4240fd450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:12:15.447 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:12:15.444+0000 7fe42bf74700 1 --2- 172.21.15.29:0/2973198699 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe4240fd040 0x7fe4240fd450 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fe4180048c0 tx=0x7fe418009e30 comp rx=0 tx=0).stop 2024-03-18T01:12:15.447 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:12:15.444+0000 7fe42bf74700 1 -- 172.21.15.29:0/2973198699 shutdown_connections 2024-03-18T01:12:15.447 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:12:15.444+0000 7fe42bf74700 1 --2- 172.21.15.29:0/2973198699 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe4240fd040 0x7fe4240fd450 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:12:15.447 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:12:15.444+0000 7fe42bf74700 1 -- 172.21.15.29:0/2973198699 >> 172.21.15.29:0/2973198699 conn(0x7fe4240f70c0 msgr2=0x7fe4240f94f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:12:15.447 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:12:15.445+0000 7fe42bf74700 1 -- 172.21.15.29:0/2973198699 shutdown_connections 2024-03-18T01:12:15.447 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:12:15.445+0000 7fe42bf74700 1 -- 172.21.15.29:0/2973198699 wait complete. 2024-03-18T01:12:15.447 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:12:15.445+0000 7fe42bf74700 1 Processor -- start 2024-03-18T01:12:15.447 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:12:15.445+0000 7fe42bf74700 1 -- start start 2024-03-18T01:12:15.447 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:12:15.445+0000 7fe42bf74700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe4240fd040 0x7fe424066460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:12:15.447 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:12:15.445+0000 7fe42bf74700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe42410fdc0 con 0x7fe4240fd040 2024-03-18T01:12:15.447 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:12:15.445+0000 7fe429d10700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe4240fd040 0x7fe424066460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:12:15.448 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:12:15.445+0000 7fe429d10700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe4240fd040 0x7fe424066460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60240/0 (socket says 172.21.15.29:60240) 2024-03-18T01:12:15.448 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:12:15.445+0000 7fe429d10700 1 -- 172.21.15.29:0/2950893115 learned_addr learned my addr 172.21.15.29:0/2950893115 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-18T01:12:15.448 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:12:15.446+0000 7fe429d10700 1 -- 172.21.15.29:0/2950893115 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe418005040 con 0x7fe4240fd040 2024-03-18T01:12:15.448 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:12:15.446+0000 7fe429d10700 1 --2- 172.21.15.29:0/2950893115 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe4240fd040 0x7fe424066460 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fe418009680 tx=0x7fe418009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:12:15.448 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:12:15.446+0000 7fe4167fc700 1 -- 172.21.15.29:0/2950893115 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe41800b5c0 con 0x7fe4240fd040 2024-03-18T01:12:15.448 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:12:15.446+0000 7fe4167fc700 1 -- 172.21.15.29:0/2950893115 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe41800be70 con 0x7fe4240fd040 2024-03-18T01:12:15.448 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:12:15.446+0000 7fe42bf74700 1 -- 172.21.15.29:0/2950893115 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe42410ff00 con 0x7fe4240fd040 2024-03-18T01:12:15.448 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:12:15.446+0000 7fe4167fc700 1 -- 172.21.15.29:0/2950893115 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe4180135a0 con 0x7fe4240fd040 2024-03-18T01:12:15.448 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:12:15.446+0000 7fe42bf74700 1 -- 172.21.15.29:0/2950893115 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4241103f0 con 0x7fe4240fd040 2024-03-18T01:12:15.449 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:12:15.447+0000 7fe4167fc700 1 -- 172.21.15.29:0/2950893115 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 13) v1 ==== 43424+0+0 (secure 0 0 0) 0x7fe418013860 con 0x7fe4240fd040 2024-03-18T01:12:15.449 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:12:15.447+0000 7fe42bf74700 1 -- 172.21.15.29:0/2950893115 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe424105910 con 0x7fe4240fd040 2024-03-18T01:12:15.449 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:12:15.447+0000 7fe4167fc700 1 --2- 172.21.15.29:0/2950893115 >> [v2:172.21.15.29:6800/2687730237,v1:172.21.15.29:6801/2687730237] conn(0x7fe410036880 0x7fe410038d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:12:15.449 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:12:15.447+0000 7fe4167fc700 1 -- 172.21.15.29:0/2950893115 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fe418049380 con 0x7fe4240fd040 2024-03-18T01:12:15.449 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:12:15.448+0000 7fe42950f700 1 --2- 172.21.15.29:0/2950893115 >> [v2:172.21.15.29:6800/2687730237,v1:172.21.15.29:6801/2687730237] conn(0x7fe410036880 0x7fe410038d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:12:15.449 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:12:15.448+0000 7fe42950f700 1 --2- 172.21.15.29:0/2950893115 >> [v2:172.21.15.29:6800/2687730237,v1:172.21.15.29:6801/2687730237] conn(0x7fe410036880 0x7fe410038d30 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fe420009990 tx=0x7fe420006e40 comp rx=0 tx=0).ready entity=mgr.14158 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:12:15.453 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:12:15.452+0000 7fe4167fc700 1 -- 172.21.15.29:0/2950893115 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175520 (secure 0 0 0) 0x7fe418011960 con 0x7fe4240fd040 2024-03-18T01:12:15.655 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:12:15.654+0000 7fe42bf74700 1 -- 172.21.15.29:0/2950893115 --> [v2:172.21.15.29:6800/2687730237,v1:172.21.15.29:6801/2687730237] -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) v1 -- 0x7fe42405e400 con 0x7fe410036880 2024-03-18T01:12:15.657 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:12:15.657+0000 7fe4167fc700 1 -- 172.21.15.29:0/2950893115 <== mgr.14158 v2:172.21.15.29:6800/2687730237 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fe42405e400 con 0x7fe410036880 2024-03-18T01:12:15.659 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:12:15.659+0000 7fe42bf74700 1 -- 172.21.15.29:0/2950893115 >> [v2:172.21.15.29:6800/2687730237,v1:172.21.15.29:6801/2687730237] conn(0x7fe410036880 msgr2=0x7fe410038d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:12:15.660 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:12:15.659+0000 7fe42bf74700 1 --2- 172.21.15.29:0/2950893115 >> [v2:172.21.15.29:6800/2687730237,v1:172.21.15.29:6801/2687730237] conn(0x7fe410036880 0x7fe410038d30 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fe420009990 tx=0x7fe420006e40 comp rx=0 tx=0).stop 2024-03-18T01:12:15.660 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:12:15.659+0000 7fe42bf74700 1 -- 172.21.15.29:0/2950893115 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe4240fd040 msgr2=0x7fe424066460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:12:15.660 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:12:15.659+0000 7fe42bf74700 1 --2- 172.21.15.29:0/2950893115 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe4240fd040 0x7fe424066460 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fe418009680 tx=0x7fe418009e30 comp rx=0 tx=0).stop 2024-03-18T01:12:15.660 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:12:15.659+0000 7fe42bf74700 1 -- 172.21.15.29:0/2950893115 shutdown_connections 2024-03-18T01:12:15.660 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:12:15.659+0000 7fe42bf74700 1 --2- 172.21.15.29:0/2950893115 >> [v2:172.21.15.29:6800/2687730237,v1:172.21.15.29:6801/2687730237] conn(0x7fe410036880 0x7fe410038d30 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:12:15.660 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:12:15.659+0000 7fe42bf74700 1 --2- 172.21.15.29:0/2950893115 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe4240fd040 0x7fe424066460 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:12:15.660 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:12:15.659+0000 7fe42bf74700 1 -- 172.21.15.29:0/2950893115 >> 172.21.15.29:0/2950893115 conn(0x7fe4240f70c0 msgr2=0x7fe424102a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:12:15.660 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:12:15.659+0000 7fe42bf74700 1 -- 172.21.15.29:0/2950893115 shutdown_connections 2024-03-18T01:12:15.660 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:12:15.659+0000 7fe42bf74700 1 -- 172.21.15.29:0/2950893115 wait complete. 2024-03-18T01:12:16.309 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi090 2024-03-18T01:12:16.310 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-03-18T01:12:16.310 DEBUG:teuthology.orchestra.run.smithi090:> dd of=/etc/ceph/ceph.conf 2024-03-18T01:12:16.327 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-03-18T01:12:16.327 DEBUG:teuthology.orchestra.run.smithi090:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-18T01:12:16.385 INFO:tasks.cephadm:Adding host smithi090 to orchestrator... 2024-03-18T01:12:16.385 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 37f51182-e4c4-11ee-95c9-87774f69a715 -- ceph orch host add smithi090 2024-03-18T01:12:16.660 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/37f51182-e4c4-11ee-95c9-87774f69a715/mon.smithi029/config 2024-03-18T01:12:16.818 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:12:16 smithi029 ceph-mon[29331]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T01:12:16.819 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:12:16 smithi029 ceph-mon[29331]: from='mgr.14158 172.21.15.29:0/4059439045' entity='mgr.smithi029.qviwhl' 2024-03-18T01:12:16.819 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:12:16 smithi029 ceph-mon[29331]: from='mgr.14158 172.21.15.29:0/4059439045' entity='mgr.smithi029.qviwhl' 2024-03-18T01:12:16.819 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:12:16 smithi029 ceph-mon[29331]: from='mgr.14158 172.21.15.29:0/4059439045' entity='mgr.smithi029.qviwhl' 2024-03-18T01:12:16.819 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:12:16 smithi029 ceph-mon[29331]: from='mgr.14158 172.21.15.29:0/4059439045' entity='mgr.smithi029.qviwhl' 2024-03-18T01:12:16.819 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:12:16 smithi029 ceph-mon[29331]: from='mgr.14158 172.21.15.29:0/4059439045' entity='mgr.smithi029.qviwhl' 2024-03-18T01:12:17.810 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:12:17 smithi029 ceph-mon[29331]: Deploying daemon node-exporter.smithi029 on smithi029 2024-03-18T01:12:18.506 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:12:18.504+0000 7fe9731ec700 1 -- 172.21.15.29:0/3628165390 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe96c0fd050 msgr2=0x7fe96c0fd460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:12:18.506 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:12:18.504+0000 7fe9731ec700 1 --2- 172.21.15.29:0/3628165390 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe96c0fd050 0x7fe96c0fd460 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fe95c0048c0 tx=0x7fe95c009e30 comp rx=0 tx=0).stop 2024-03-18T01:12:18.506 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:12:18.504+0000 7fe9731ec700 1 -- 172.21.15.29:0/3628165390 shutdown_connections 2024-03-18T01:12:18.506 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:12:18.504+0000 7fe9731ec700 1 --2- 172.21.15.29:0/3628165390 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe96c0fd050 0x7fe96c0fd460 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:12:18.507 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:12:18.504+0000 7fe9731ec700 1 -- 172.21.15.29:0/3628165390 >> 172.21.15.29:0/3628165390 conn(0x7fe96c0f70d0 msgr2=0x7fe96c0f9500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:12:18.507 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:12:18.504+0000 7fe9731ec700 1 -- 172.21.15.29:0/3628165390 shutdown_connections 2024-03-18T01:12:18.507 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:12:18.504+0000 7fe9731ec700 1 -- 172.21.15.29:0/3628165390 wait complete. 2024-03-18T01:12:18.507 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:12:18.505+0000 7fe9731ec700 1 Processor -- start 2024-03-18T01:12:18.507 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:12:18.505+0000 7fe9731ec700 1 -- start start 2024-03-18T01:12:18.507 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:12:18.505+0000 7fe9731ec700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe96c0fd050 0x7fe96c065bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:12:18.507 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:12:18.505+0000 7fe9731ec700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe96c10fdc0 con 0x7fe96c0fd050 2024-03-18T01:12:18.507 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:12:18.505+0000 7fe970f88700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe96c0fd050 0x7fe96c065bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:12:18.507 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:12:18.505+0000 7fe970f88700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe96c0fd050 0x7fe96c065bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60272/0 (socket says 172.21.15.29:60272) 2024-03-18T01:12:18.507 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:12:18.505+0000 7fe970f88700 1 -- 172.21.15.29:0/2525784026 learned_addr learned my addr 172.21.15.29:0/2525784026 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-18T01:12:18.507 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:12:18.505+0000 7fe970f88700 1 -- 172.21.15.29:0/2525784026 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe95c005040 con 0x7fe96c0fd050 2024-03-18T01:12:18.507 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:12:18.506+0000 7fe970f88700 1 --2- 172.21.15.29:0/2525784026 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe96c0fd050 0x7fe96c065bf0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fe95c0096b0 tx=0x7fe95c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:12:18.507 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:12:18.506+0000 7fe9697fa700 1 -- 172.21.15.29:0/2525784026 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe95c00bed0 con 0x7fe96c0fd050 2024-03-18T01:12:18.507 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:12:18.506+0000 7fe9697fa700 1 -- 172.21.15.29:0/2525784026 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe95c00b5c0 con 0x7fe96c0fd050 2024-03-18T01:12:18.508 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:12:18.506+0000 7fe9731ec700 1 -- 172.21.15.29:0/2525784026 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe96c10ff00 con 0x7fe96c0fd050 2024-03-18T01:12:18.508 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:12:18.506+0000 7fe9697fa700 1 -- 172.21.15.29:0/2525784026 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe95c01c5b0 con 0x7fe96c0fd050 2024-03-18T01:12:18.508 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:12:18.506+0000 7fe9731ec700 1 -- 172.21.15.29:0/2525784026 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe96c1103f0 con 0x7fe96c0fd050 2024-03-18T01:12:18.509 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:12:18.507+0000 7fe9697fa700 1 -- 172.21.15.29:0/2525784026 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 13) v1 ==== 43424+0+0 (secure 0 0 0) 0x7fe95c01c820 con 0x7fe96c0fd050 2024-03-18T01:12:18.509 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:12:18.507+0000 7fe9697fa700 1 --2- 172.21.15.29:0/2525784026 >> [v2:172.21.15.29:6800/2687730237,v1:172.21.15.29:6801/2687730237] conn(0x7fe954036880 0x7fe954038d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:12:18.509 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:12:18.507+0000 7fe9697fa700 1 -- 172.21.15.29:0/2525784026 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fe95c049c50 con 0x7fe96c0fd050 2024-03-18T01:12:18.509 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:12:18.507+0000 7fe9731ec700 1 -- 172.21.15.29:0/2525784026 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe96c105920 con 0x7fe96c0fd050 2024-03-18T01:12:18.509 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:12:18.507+0000 7fe96bfff700 1 --2- 172.21.15.29:0/2525784026 >> [v2:172.21.15.29:6800/2687730237,v1:172.21.15.29:6801/2687730237] conn(0x7fe954036880 0x7fe954038d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:12:18.509 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:12:18.507+0000 7fe96bfff700 1 --2- 172.21.15.29:0/2525784026 >> [v2:172.21.15.29:6800/2687730237,v1:172.21.15.29:6801/2687730237] conn(0x7fe954036880 0x7fe954038d30 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fe960009990 tx=0x7fe960006e40 comp rx=0 tx=0).ready entity=mgr.14158 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:12:18.512 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:12:18.512+0000 7fe9697fa700 1 -- 172.21.15.29:0/2525784026 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175520 (secure 0 0 0) 0x7fe95c023300 con 0x7fe96c0fd050 2024-03-18T01:12:18.747 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:12:18.746+0000 7fe9731ec700 1 -- 172.21.15.29:0/2525784026 --> [v2:172.21.15.29:6800/2687730237,v1:172.21.15.29:6801/2687730237] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi090", "target": ["mon-mgr", ""]}) v1 -- 0x7fe96c04bed0 con 0x7fe954036880 2024-03-18T01:12:20.505 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:12:20 smithi029 ceph-mon[29331]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi090", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T01:12:20.505 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:12:20 smithi029 ceph-mon[29331]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T01:12:20.505 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:12:20 smithi029 ceph-mon[29331]: from='mgr.14158 172.21.15.29:0/4059439045' entity='mgr.smithi029.qviwhl' 2024-03-18T01:12:20.505 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:12:20 smithi029 ceph-mon[29331]: Deploying cephadm binary to smithi090 2024-03-18T01:12:20.885 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:12:20.884+0000 7fe9697fa700 1 -- 172.21.15.29:0/2525784026 <== mgr.14158 v2:172.21.15.29:6800/2687730237 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+48 (secure 0 0 0) 0x7fe96c04bed0 con 0x7fe954036880 2024-03-18T01:12:20.885 INFO:teuthology.orchestra.run.smithi029.stdout:Added host 'smithi090' with addr '172.21.15.90' 2024-03-18T01:12:20.887 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:12:20.887+0000 7fe9731ec700 1 -- 172.21.15.29:0/2525784026 >> [v2:172.21.15.29:6800/2687730237,v1:172.21.15.29:6801/2687730237] conn(0x7fe954036880 msgr2=0x7fe954038d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:12:20.887 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:12:20.887+0000 7fe9731ec700 1 --2- 172.21.15.29:0/2525784026 >> [v2:172.21.15.29:6800/2687730237,v1:172.21.15.29:6801/2687730237] conn(0x7fe954036880 0x7fe954038d30 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fe960009990 tx=0x7fe960006e40 comp rx=0 tx=0).stop 2024-03-18T01:12:20.887 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:12:20.887+0000 7fe9731ec700 1 -- 172.21.15.29:0/2525784026 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe96c0fd050 msgr2=0x7fe96c065bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:12:20.887 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:12:20.887+0000 7fe9731ec700 1 --2- 172.21.15.29:0/2525784026 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe96c0fd050 0x7fe96c065bf0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fe95c0096b0 tx=0x7fe95c009e30 comp rx=0 tx=0).stop 2024-03-18T01:12:20.887 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:12:20.887+0000 7fe9731ec700 1 -- 172.21.15.29:0/2525784026 shutdown_connections 2024-03-18T01:12:20.887 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:12:20.887+0000 7fe9731ec700 1 --2- 172.21.15.29:0/2525784026 >> [v2:172.21.15.29:6800/2687730237,v1:172.21.15.29:6801/2687730237] conn(0x7fe954036880 0x7fe954038d30 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:12:20.887 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:12:20.887+0000 7fe9731ec700 1 --2- 172.21.15.29:0/2525784026 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe96c0fd050 0x7fe96c065bf0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:12:20.887 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:12:20.887+0000 7fe9731ec700 1 -- 172.21.15.29:0/2525784026 >> 172.21.15.29:0/2525784026 conn(0x7fe96c0f70d0 msgr2=0x7fe96c102aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:12:20.888 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:12:20.887+0000 7fe9731ec700 1 -- 172.21.15.29:0/2525784026 shutdown_connections 2024-03-18T01:12:20.888 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:12:20.887+0000 7fe9731ec700 1 -- 172.21.15.29:0/2525784026 wait complete. 2024-03-18T01:12:21.651 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 37f51182-e4c4-11ee-95c9-87774f69a715 -- ceph orch host ls --format=json 2024-03-18T01:12:22.173 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/37f51182-e4c4-11ee-95c9-87774f69a715/mon.smithi029/config 2024-03-18T01:12:22.298 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:12:21 smithi029 ceph-mon[29331]: from='mgr.14158 172.21.15.29:0/4059439045' entity='mgr.smithi029.qviwhl' 2024-03-18T01:12:22.298 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:12:21 smithi029 ceph-mon[29331]: Added host smithi090 2024-03-18T01:12:22.298 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:12:21 smithi029 ceph-mon[29331]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T01:12:23.319 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:12:23 smithi029 ceph-mon[29331]: from='mgr.14158 172.21.15.29:0/4059439045' entity='mgr.smithi029.qviwhl' 2024-03-18T01:12:23.319 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:12:23 smithi029 ceph-mon[29331]: from='mgr.14158 172.21.15.29:0/4059439045' entity='mgr.smithi029.qviwhl' 2024-03-18T01:12:23.319 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:12:23 smithi029 ceph-mon[29331]: from='mgr.14158 172.21.15.29:0/4059439045' entity='mgr.smithi029.qviwhl' 2024-03-18T01:12:23.319 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:12:23 smithi029 ceph-mon[29331]: from='mgr.14158 172.21.15.29:0/4059439045' entity='mgr.smithi029.qviwhl' 2024-03-18T01:12:23.319 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:12:23 smithi029 ceph-mon[29331]: Deploying daemon alertmanager.smithi029 on smithi029 2024-03-18T01:12:23.893 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:12:23.891+0000 7f9e4cba6700 1 -- 172.21.15.29:0/4080984035 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9e480fd050 msgr2=0x7f9e480fd460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:12:23.893 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:12:23.891+0000 7f9e4cba6700 1 --2- 172.21.15.29:0/4080984035 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9e480fd050 0x7f9e480fd460 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f9e38009c60 tx=0x7f9e38004d10 comp rx=0 tx=0).stop 2024-03-18T01:12:23.893 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:12:23.891+0000 7f9e4cba6700 1 -- 172.21.15.29:0/4080984035 shutdown_connections 2024-03-18T01:12:23.893 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:12:23.891+0000 7f9e4cba6700 1 --2- 172.21.15.29:0/4080984035 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9e480fd050 0x7f9e480fd460 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:12:23.893 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:12:23.891+0000 7f9e4cba6700 1 -- 172.21.15.29:0/4080984035 >> 172.21.15.29:0/4080984035 conn(0x7f9e480f70d0 msgr2=0x7f9e480f9500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:12:23.893 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:12:23.891+0000 7f9e4cba6700 1 -- 172.21.15.29:0/4080984035 shutdown_connections 2024-03-18T01:12:23.894 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:12:23.891+0000 7f9e4cba6700 1 -- 172.21.15.29:0/4080984035 wait complete. 2024-03-18T01:12:23.894 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:12:23.891+0000 7f9e4cba6700 1 Processor -- start 2024-03-18T01:12:23.894 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:12:23.891+0000 7f9e4cba6700 1 -- start start 2024-03-18T01:12:23.894 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:12:23.892+0000 7f9e4cba6700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9e480fd050 0x7f9e4810f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:12:23.894 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:12:23.892+0000 7f9e4cba6700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e4810f810 con 0x7f9e480fd050 2024-03-18T01:12:23.894 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:12:23.892+0000 7f9e4659c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9e480fd050 0x7f9e4810f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:12:23.894 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:12:23.892+0000 7f9e4659c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9e480fd050 0x7f9e4810f2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33844/0 (socket says 172.21.15.29:33844) 2024-03-18T01:12:23.894 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:12:23.892+0000 7f9e4659c700 1 -- 172.21.15.29:0/2898511308 learned_addr learned my addr 172.21.15.29:0/2898511308 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-18T01:12:23.894 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:12:23.892+0000 7f9e4659c700 1 -- 172.21.15.29:0/2898511308 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e38009940 con 0x7f9e480fd050 2024-03-18T01:12:23.894 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:12:23.892+0000 7f9e4659c700 1 --2- 172.21.15.29:0/2898511308 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9e480fd050 0x7f9e4810f2d0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f9e38004e60 tx=0x7f9e38004d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:12:23.894 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:12:23.892+0000 7f9e3effd700 1 -- 172.21.15.29:0/2898511308 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9e38012410 con 0x7f9e480fd050 2024-03-18T01:12:23.894 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:12:23.892+0000 7f9e3effd700 1 -- 172.21.15.29:0/2898511308 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9e38012a40 con 0x7f9e480fd050 2024-03-18T01:12:23.894 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:12:23.892+0000 7f9e4cba6700 1 -- 172.21.15.29:0/2898511308 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e4810fa10 con 0x7f9e480fd050 2024-03-18T01:12:23.894 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:12:23.892+0000 7f9e3effd700 1 -- 172.21.15.29:0/2898511308 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9e3801ba40 con 0x7f9e480fd050 2024-03-18T01:12:23.895 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:12:23.892+0000 7f9e4cba6700 1 -- 172.21.15.29:0/2898511308 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e48110580 con 0x7f9e480fd050 2024-03-18T01:12:23.895 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:12:23.893+0000 7f9e3effd700 1 -- 172.21.15.29:0/2898511308 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 13) v1 ==== 43424+0+0 (secure 0 0 0) 0x7f9e38019040 con 0x7f9e480fd050 2024-03-18T01:12:23.895 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:12:23.893+0000 7f9e4cba6700 1 -- 172.21.15.29:0/2898511308 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e48101e70 con 0x7f9e480fd050 2024-03-18T01:12:23.896 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:12:23.893+0000 7f9e3effd700 1 --2- 172.21.15.29:0/2898511308 >> [v2:172.21.15.29:6800/2687730237,v1:172.21.15.29:6801/2687730237] conn(0x7f9e34036880 0x7f9e34038d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:12:23.896 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:12:23.893+0000 7f9e3effd700 1 -- 172.21.15.29:0/2898511308 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f9e38049c00 con 0x7f9e480fd050 2024-03-18T01:12:23.896 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:12:23.893+0000 7f9e45d9b700 1 --2- 172.21.15.29:0/2898511308 >> [v2:172.21.15.29:6800/2687730237,v1:172.21.15.29:6801/2687730237] conn(0x7f9e34036880 0x7f9e34038d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:12:23.896 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:12:23.894+0000 7f9e45d9b700 1 --2- 172.21.15.29:0/2898511308 >> [v2:172.21.15.29:6800/2687730237,v1:172.21.15.29:6801/2687730237] conn(0x7f9e34036880 0x7f9e34038d30 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f9e30009990 tx=0x7f9e30006e40 comp rx=0 tx=0).ready entity=mgr.14158 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:12:23.898 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:12:23.898+0000 7f9e3effd700 1 -- 172.21.15.29:0/2898511308 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175520 (secure 0 0 0) 0x7f9e38050080 con 0x7f9e480fd050 2024-03-18T01:12:24.098 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:12:24.097+0000 7f9e4cba6700 1 -- 172.21.15.29:0/2898511308 --> [v2:172.21.15.29:6800/2687730237,v1:172.21.15.29:6801/2687730237] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f9e4805f690 con 0x7f9e34036880 2024-03-18T01:12:24.099 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:12:24.099+0000 7f9e3effd700 1 -- 172.21.15.29:0/2898511308 <== mgr.14158 v2:172.21.15.29:6800/2687730237 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+159 (secure 0 0 0) 0x7f9e4805f690 con 0x7f9e34036880 2024-03-18T01:12:24.099 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-18T01:12:24.099 INFO:teuthology.orchestra.run.smithi029.stdout:[{"addr": "172.21.15.29", "hostname": "smithi029", "labels": [], "status": ""}, {"addr": "172.21.15.90", "hostname": "smithi090", "labels": [], "status": ""}] 2024-03-18T01:12:24.101 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:12:24.100+0000 7f9e4cba6700 1 -- 172.21.15.29:0/2898511308 >> [v2:172.21.15.29:6800/2687730237,v1:172.21.15.29:6801/2687730237] conn(0x7f9e34036880 msgr2=0x7f9e34038d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:12:24.101 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:12:24.100+0000 7f9e4cba6700 1 --2- 172.21.15.29:0/2898511308 >> [v2:172.21.15.29:6800/2687730237,v1:172.21.15.29:6801/2687730237] conn(0x7f9e34036880 0x7f9e34038d30 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f9e30009990 tx=0x7f9e30006e40 comp rx=0 tx=0).stop 2024-03-18T01:12:24.101 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:12:24.100+0000 7f9e4cba6700 1 -- 172.21.15.29:0/2898511308 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9e480fd050 msgr2=0x7f9e4810f2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:12:24.101 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:12:24.100+0000 7f9e4cba6700 1 --2- 172.21.15.29:0/2898511308 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9e480fd050 0x7f9e4810f2d0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f9e38004e60 tx=0x7f9e38004d10 comp rx=0 tx=0).stop 2024-03-18T01:12:24.101 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:12:24.101+0000 7f9e4cba6700 1 -- 172.21.15.29:0/2898511308 shutdown_connections 2024-03-18T01:12:24.102 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:12:24.101+0000 7f9e4cba6700 1 --2- 172.21.15.29:0/2898511308 >> [v2:172.21.15.29:6800/2687730237,v1:172.21.15.29:6801/2687730237] conn(0x7f9e34036880 0x7f9e34038d30 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:12:24.102 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:12:24.101+0000 7f9e4cba6700 1 --2- 172.21.15.29:0/2898511308 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9e480fd050 0x7f9e4810f2d0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:12:24.102 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:12:24.101+0000 7f9e4cba6700 1 -- 172.21.15.29:0/2898511308 >> 172.21.15.29:0/2898511308 conn(0x7f9e480f70d0 msgr2=0x7f9e480f9500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:12:24.102 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:12:24.101+0000 7f9e4cba6700 1 -- 172.21.15.29:0/2898511308 shutdown_connections 2024-03-18T01:12:24.102 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:12:24.101+0000 7f9e4cba6700 1 -- 172.21.15.29:0/2898511308 wait complete. 2024-03-18T01:12:24.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:12:24 smithi029 ceph-mon[29331]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T01:12:25.068 INFO:tasks.cephadm:Setting crush tunables to default 2024-03-18T01:12:25.068 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 37f51182-e4c4-11ee-95c9-87774f69a715 -- ceph osd crush tunables default 2024-03-18T01:12:25.318 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:12:25 smithi029 ceph-mon[29331]: from='client.14186 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-18T01:12:25.319 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:12:25 smithi029 ceph-mon[29331]: from='mgr.14158 172.21.15.29:0/4059439045' entity='mgr.smithi029.qviwhl' 2024-03-18T01:12:25.327 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/37f51182-e4c4-11ee-95c9-87774f69a715/mon.smithi029/config 2024-03-18T01:12:26.568 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:12:26 smithi029 ceph-mon[29331]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T01:12:28.538 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:12:28 smithi029 ceph-mon[29331]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T01:12:28.611 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:12:28.538+0000 7f694400b700 1 -- 172.21.15.29:0/1579132198 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f693c0fca50 msgr2=0x7f693c0fce60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:12:28.611 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:12:28.538+0000 7f694400b700 1 --2- 172.21.15.29:0/1579132198 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f693c0fca50 0x7f693c0fce60 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f69380048c0 tx=0x7f6938009e30 comp rx=0 tx=0).stop 2024-03-18T01:12:28.611 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:12:28.538+0000 7f694400b700 1 -- 172.21.15.29:0/1579132198 shutdown_connections 2024-03-18T01:12:28.611 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:12:28.538+0000 7f694400b700 1 --2- 172.21.15.29:0/1579132198 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f693c0fca50 0x7f693c0fce60 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:12:28.612 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:12:28.538+0000 7f694400b700 1 -- 172.21.15.29:0/1579132198 >> 172.21.15.29:0/1579132198 conn(0x7f693c0f8190 msgr2=0x7f693c0fa5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:12:28.612 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:12:28.538+0000 7f694400b700 1 -- 172.21.15.29:0/1579132198 shutdown_connections 2024-03-18T01:12:28.612 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:12:28.538+0000 7f694400b700 1 -- 172.21.15.29:0/1579132198 wait complete. 2024-03-18T01:12:28.612 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:12:28.538+0000 7f694400b700 1 Processor -- start 2024-03-18T01:12:28.612 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:12:28.538+0000 7f694400b700 1 -- start start 2024-03-18T01:12:28.612 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:12:28.539+0000 7f694400b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f693c0fca50 0x7f693c10d0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:12:28.612 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:12:28.539+0000 7f694400b700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f693c10d5f0 con 0x7f693c0fca50 2024-03-18T01:12:28.612 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:12:28.539+0000 7f6941da7700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f693c0fca50 0x7f693c10d0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:12:28.612 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:12:28.539+0000 7f6941da7700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f693c0fca50 0x7f693c10d0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33858/0 (socket says 172.21.15.29:33858) 2024-03-18T01:12:28.612 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:12:28.539+0000 7f6941da7700 1 -- 172.21.15.29:0/3607485839 learned_addr learned my addr 172.21.15.29:0/3607485839 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-18T01:12:28.612 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:12:28.539+0000 7f6941da7700 1 -- 172.21.15.29:0/3607485839 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6938005040 con 0x7f693c0fca50 2024-03-18T01:12:28.612 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:12:28.539+0000 7f6941da7700 1 --2- 172.21.15.29:0/3607485839 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f693c0fca50 0x7f693c10d0b0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f69380096b0 tx=0x7f6938009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:12:28.613 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:12:28.539+0000 7f69327fc700 1 -- 172.21.15.29:0/3607485839 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f693800bed0 con 0x7f693c0fca50 2024-03-18T01:12:28.613 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:12:28.539+0000 7f69327fc700 1 -- 172.21.15.29:0/3607485839 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f693800b5c0 con 0x7f693c0fca50 2024-03-18T01:12:28.613 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:12:28.539+0000 7f694400b700 1 -- 172.21.15.29:0/3607485839 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f693c10d7f0 con 0x7f693c0fca50 2024-03-18T01:12:28.613 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:12:28.539+0000 7f69327fc700 1 -- 172.21.15.29:0/3607485839 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f693801c5b0 con 0x7f693c0fca50 2024-03-18T01:12:28.613 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:12:28.539+0000 7f694400b700 1 -- 172.21.15.29:0/3607485839 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f693c10e360 con 0x7f693c0fca50 2024-03-18T01:12:28.613 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:12:28.540+0000 7f69327fc700 1 -- 172.21.15.29:0/3607485839 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 13) v1 ==== 43424+0+0 (secure 0 0 0) 0x7f6938013a80 con 0x7f693c0fca50 2024-03-18T01:12:28.613 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:12:28.540+0000 7f69327fc700 1 --2- 172.21.15.29:0/3607485839 >> [v2:172.21.15.29:6800/2687730237,v1:172.21.15.29:6801/2687730237] conn(0x7f6928036880 0x7f6928038d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:12:28.613 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:12:28.540+0000 7f694400b700 1 -- 172.21.15.29:0/3607485839 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f693c102f60 con 0x7f693c0fca50 2024-03-18T01:12:28.613 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:12:28.540+0000 7f69327fc700 1 -- 172.21.15.29:0/3607485839 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f6938049a10 con 0x7f693c0fca50 2024-03-18T01:12:28.613 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:12:28.540+0000 7f69415a6700 1 --2- 172.21.15.29:0/3607485839 >> [v2:172.21.15.29:6800/2687730237,v1:172.21.15.29:6801/2687730237] conn(0x7f6928036880 0x7f6928038d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:12:28.613 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:12:28.541+0000 7f69415a6700 1 --2- 172.21.15.29:0/3607485839 >> [v2:172.21.15.29:6800/2687730237,v1:172.21.15.29:6801/2687730237] conn(0x7f6928036880 0x7f6928038d30 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f692c009990 tx=0x7f692c006e40 comp rx=0 tx=0).ready entity=mgr.14158 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:12:28.613 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:12:28.546+0000 7f69327fc700 1 -- 172.21.15.29:0/3607485839 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175520 (secure 0 0 0) 0x7f6938011550 con 0x7f693c0fca50 2024-03-18T01:12:28.746 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:12:28.746+0000 7f694400b700 1 -- 172.21.15.29:0/3607485839 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f693c10ee30 con 0x7f693c0fca50 2024-03-18T01:12:29.228 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:12:29.227+0000 7f69327fc700 1 -- 172.21.15.29:0/3607485839 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) v1 ==== 124+0+0 (secure 0 0 0) 0x7f693802b6d0 con 0x7f693c0fca50 2024-03-18T01:12:29.230 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:12:29.230+0000 7f694400b700 1 -- 172.21.15.29:0/3607485839 >> [v2:172.21.15.29:6800/2687730237,v1:172.21.15.29:6801/2687730237] conn(0x7f6928036880 msgr2=0x7f6928038d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:12:29.230 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:12:29.230+0000 7f694400b700 1 --2- 172.21.15.29:0/3607485839 >> [v2:172.21.15.29:6800/2687730237,v1:172.21.15.29:6801/2687730237] conn(0x7f6928036880 0x7f6928038d30 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f692c009990 tx=0x7f692c006e40 comp rx=0 tx=0).stop 2024-03-18T01:12:29.230 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:12:29.230+0000 7f694400b700 1 -- 172.21.15.29:0/3607485839 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f693c0fca50 msgr2=0x7f693c10d0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:12:29.230 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:12:29.230+0000 7f694400b700 1 --2- 172.21.15.29:0/3607485839 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f693c0fca50 0x7f693c10d0b0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f69380096b0 tx=0x7f6938009e30 comp rx=0 tx=0).stop 2024-03-18T01:12:29.230 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:12:29.230+0000 7f694400b700 1 -- 172.21.15.29:0/3607485839 shutdown_connections 2024-03-18T01:12:29.231 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:12:29.230+0000 7f694400b700 1 --2- 172.21.15.29:0/3607485839 >> [v2:172.21.15.29:6800/2687730237,v1:172.21.15.29:6801/2687730237] conn(0x7f6928036880 0x7f6928038d30 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:12:29.231 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:12:29.230+0000 7f694400b700 1 --2- 172.21.15.29:0/3607485839 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f693c0fca50 0x7f693c10d0b0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:12:29.231 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:12:29.230+0000 7f694400b700 1 -- 172.21.15.29:0/3607485839 >> 172.21.15.29:0/3607485839 conn(0x7f693c0f8190 msgr2=0x7f693c103b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:12:29.231 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:12:29.230+0000 7f694400b700 1 -- 172.21.15.29:0/3607485839 shutdown_connections 2024-03-18T01:12:29.231 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:12:29.230+0000 7f694400b700 1 -- 172.21.15.29:0/3607485839 wait complete. 2024-03-18T01:12:29.231 INFO:teuthology.orchestra.run.smithi029.stderr:adjusted tunables profile to default 2024-03-18T01:12:29.513 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:12:29 smithi029 ceph-mon[29331]: from='client.? 172.21.15.29:0/3607485839' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-18T01:12:30.403 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:12:30 smithi029 ceph-mon[29331]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T01:12:30.403 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:12:30 smithi029 ceph-mon[29331]: from='client.? 172.21.15.29:0/3607485839' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-18T01:12:30.403 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:12:30 smithi029 ceph-mon[29331]: osdmap e4: 0 total, 0 up, 0 in 2024-03-18T01:12:32.119 INFO:tasks.cephadm:Adding mon.smithi029 on smithi029 2024-03-18T01:12:32.119 INFO:tasks.cephadm:Adding mon.smithi090 on smithi090 2024-03-18T01:12:32.119 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 37f51182-e4c4-11ee-95c9-87774f69a715 -- ceph orch apply mon '2;smithi029:172.21.15.29=smithi029;smithi090:172.21.15.90=smithi090' 2024-03-18T01:12:32.366 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-18T01:12:32.412 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-18T01:12:32.553 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:12:32 smithi029 ceph-mon[29331]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T01:12:33.915 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:33.915+0000 7f5c5aa3a700 1 -- 172.21.15.90:0/762011503 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5c540669e0 msgr2=0x7f5c540ff690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:12:33.915 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:33.915+0000 7f5c5aa3a700 1 --2- 172.21.15.90:0/762011503 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5c540669e0 0x7f5c540ff690 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f5c440048c0 tx=0x7f5c44009e30 comp rx=0 tx=0).stop 2024-03-18T01:12:33.916 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:33.915+0000 7f5c5aa3a700 1 -- 172.21.15.90:0/762011503 shutdown_connections 2024-03-18T01:12:33.916 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:33.915+0000 7f5c5aa3a700 1 --2- 172.21.15.90:0/762011503 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5c540669e0 0x7f5c540ff690 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:12:33.916 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:33.915+0000 7f5c5aa3a700 1 -- 172.21.15.90:0/762011503 >> 172.21.15.90:0/762011503 conn(0x7f5c540f70c0 msgr2=0x7f5c540f94d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:12:33.916 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:33.915+0000 7f5c5aa3a700 1 -- 172.21.15.90:0/762011503 shutdown_connections 2024-03-18T01:12:33.916 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:33.915+0000 7f5c5aa3a700 1 -- 172.21.15.90:0/762011503 wait complete. 2024-03-18T01:12:33.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:33.915+0000 7f5c5aa3a700 1 Processor -- start 2024-03-18T01:12:33.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:33.916+0000 7f5c5aa3a700 1 -- start start 2024-03-18T01:12:33.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:33.916+0000 7f5c5aa3a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5c540669e0 0x7f5c5410f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:12:33.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:33.916+0000 7f5c5aa3a700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c5410f810 con 0x7f5c540669e0 2024-03-18T01:12:33.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:33.916+0000 7f5c53fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5c540669e0 0x7f5c5410f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:12:33.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:33.916+0000 7f5c53fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5c540669e0 0x7f5c5410f2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.90:36940/0 (socket says 172.21.15.90:36940) 2024-03-18T01:12:33.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:33.916+0000 7f5c53fff700 1 -- 172.21.15.90:0/993447247 learned_addr learned my addr 172.21.15.90:0/993447247 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-18T01:12:33.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:33.916+0000 7f5c53fff700 1 -- 172.21.15.90:0/993447247 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c44005040 con 0x7f5c540669e0 2024-03-18T01:12:33.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:33.916+0000 7f5c53fff700 1 --2- 172.21.15.90:0/993447247 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5c540669e0 0x7f5c5410f2d0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f5c44009680 tx=0x7f5c44009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:12:33.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:33.917+0000 7f5c50ff9700 1 -- 172.21.15.90:0/993447247 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5c4400bed0 con 0x7f5c540669e0 2024-03-18T01:12:33.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:33.917+0000 7f5c50ff9700 1 -- 172.21.15.90:0/993447247 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5c4400b5c0 con 0x7f5c540669e0 2024-03-18T01:12:33.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:33.917+0000 7f5c5aa3a700 1 -- 172.21.15.90:0/993447247 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c5410fa10 con 0x7f5c540669e0 2024-03-18T01:12:33.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:33.917+0000 7f5c50ff9700 1 -- 172.21.15.90:0/993447247 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5c4401c5b0 con 0x7f5c540669e0 2024-03-18T01:12:33.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:33.917+0000 7f5c5aa3a700 1 -- 172.21.15.90:0/993447247 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c54110580 con 0x7f5c540669e0 2024-03-18T01:12:33.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:33.917+0000 7f5c50ff9700 1 -- 172.21.15.90:0/993447247 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 13) v1 ==== 43424+0+0 (secure 0 0 0) 0x7f5c4401c710 con 0x7f5c540669e0 2024-03-18T01:12:33.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:33.918+0000 7f5c5aa3a700 1 -- 172.21.15.90:0/993447247 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c54102f40 con 0x7f5c540669e0 2024-03-18T01:12:33.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:33.918+0000 7f5c50ff9700 1 --2- 172.21.15.90:0/993447247 >> [v2:172.21.15.29:6800/2687730237,v1:172.21.15.29:6801/2687730237] conn(0x7f5c3c036880 0x7f5c3c038d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:12:33.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:33.918+0000 7f5c50ff9700 1 -- 172.21.15.90:0/993447247 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f5c440499f0 con 0x7f5c540669e0 2024-03-18T01:12:33.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:33.918+0000 7f5c537fe700 1 --2- 172.21.15.90:0/993447247 >> [v2:172.21.15.29:6800/2687730237,v1:172.21.15.29:6801/2687730237] conn(0x7f5c3c036880 0x7f5c3c038d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:12:33.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:33.918+0000 7f5c537fe700 1 --2- 172.21.15.90:0/993447247 >> [v2:172.21.15.29:6800/2687730237,v1:172.21.15.29:6801/2687730237] conn(0x7f5c3c036880 0x7f5c3c038d30 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f5c48009990 tx=0x7f5c48006e40 comp rx=0 tx=0).ready entity=mgr.14158 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:12:33.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:33.923+0000 7f5c50ff9700 1 -- 172.21.15.90:0/993447247 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175520 (secure 0 0 0) 0x7f5c44025400 con 0x7f5c540669e0 2024-03-18T01:12:34.115 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:34.114+0000 7f5c5aa3a700 1 -- 172.21.15.90:0/993447247 --> [v2:172.21.15.29:6800/2687730237,v1:172.21.15.29:6801/2687730237] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi029:172.21.15.29=smithi029;smithi090:172.21.15.90=smithi090", "target": ["mon-mgr", ""]}) v1 -- 0x7f5c540fca90 con 0x7f5c3c036880 2024-03-18T01:12:34.119 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:34.119+0000 7f5c50ff9700 1 -- 172.21.15.90:0/993447247 <== mgr.14158 v2:172.21.15.29:6800/2687730237 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f5c540fca90 con 0x7f5c3c036880 2024-03-18T01:12:34.120 INFO:teuthology.orchestra.run.smithi090.stdout:Scheduled mon update... 2024-03-18T01:12:34.121 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:34.121+0000 7f5c5aa3a700 1 -- 172.21.15.90:0/993447247 >> [v2:172.21.15.29:6800/2687730237,v1:172.21.15.29:6801/2687730237] conn(0x7f5c3c036880 msgr2=0x7f5c3c038d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:12:34.121 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:34.121+0000 7f5c5aa3a700 1 --2- 172.21.15.90:0/993447247 >> [v2:172.21.15.29:6800/2687730237,v1:172.21.15.29:6801/2687730237] conn(0x7f5c3c036880 0x7f5c3c038d30 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f5c48009990 tx=0x7f5c48006e40 comp rx=0 tx=0).stop 2024-03-18T01:12:34.121 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:34.121+0000 7f5c5aa3a700 1 -- 172.21.15.90:0/993447247 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5c540669e0 msgr2=0x7f5c5410f2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:12:34.122 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:34.121+0000 7f5c5aa3a700 1 --2- 172.21.15.90:0/993447247 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5c540669e0 0x7f5c5410f2d0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f5c44009680 tx=0x7f5c44009e30 comp rx=0 tx=0).stop 2024-03-18T01:12:34.122 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:34.121+0000 7f5c5aa3a700 1 -- 172.21.15.90:0/993447247 shutdown_connections 2024-03-18T01:12:34.122 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:34.121+0000 7f5c5aa3a700 1 --2- 172.21.15.90:0/993447247 >> [v2:172.21.15.29:6800/2687730237,v1:172.21.15.29:6801/2687730237] conn(0x7f5c3c036880 0x7f5c3c038d30 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:12:34.122 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:34.121+0000 7f5c5aa3a700 1 --2- 172.21.15.90:0/993447247 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5c540669e0 0x7f5c5410f2d0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:12:34.122 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:34.121+0000 7f5c5aa3a700 1 -- 172.21.15.90:0/993447247 >> 172.21.15.90:0/993447247 conn(0x7f5c540f70c0 msgr2=0x7f5c540f94a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:12:34.122 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:34.121+0000 7f5c5aa3a700 1 -- 172.21.15.90:0/993447247 shutdown_connections 2024-03-18T01:12:34.122 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:34.121+0000 7f5c5aa3a700 1 -- 172.21.15.90:0/993447247 wait complete. 2024-03-18T01:12:34.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:12:34 smithi029 ceph-mon[29331]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T01:12:34.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:12:34 smithi029 ceph-mon[29331]: from='mgr.14158 172.21.15.29:0/4059439045' entity='mgr.smithi029.qviwhl' 2024-03-18T01:12:34.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:12:34 smithi029 ceph-mon[29331]: from='mgr.14158 172.21.15.29:0/4059439045' entity='mgr.smithi029.qviwhl' 2024-03-18T01:12:34.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:12:34 smithi029 ceph-mon[29331]: from='mgr.14158 172.21.15.29:0/4059439045' entity='mgr.smithi029.qviwhl' 2024-03-18T01:12:34.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:12:34 smithi029 ceph-mon[29331]: from='mgr.14158 172.21.15.29:0/4059439045' entity='mgr.smithi029.qviwhl' 2024-03-18T01:12:34.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:12:34 smithi029 ceph-mon[29331]: from='mgr.14158 172.21.15.29:0/4059439045' entity='mgr.smithi029.qviwhl' 2024-03-18T01:12:34.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:12:34 smithi029 ceph-mon[29331]: from='mgr.14158 172.21.15.29:0/4059439045' entity='mgr.smithi029.qviwhl' 2024-03-18T01:12:34.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:12:34 smithi029 ceph-mon[29331]: from='mgr.14158 172.21.15.29:0/4059439045' entity='mgr.smithi029.qviwhl' 2024-03-18T01:12:34.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:12:34 smithi029 ceph-mon[29331]: from='mgr.14158 172.21.15.29:0/4059439045' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-18T01:12:34.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:12:34 smithi029 ceph-mon[29331]: from='mgr.14158 172.21.15.29:0/4059439045' entity='mgr.smithi029.qviwhl' 2024-03-18T01:12:34.570 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:12:34 smithi029 ceph-mon[29331]: from='mgr.14158 172.21.15.29:0/4059439045' entity='mgr.smithi029.qviwhl' 2024-03-18T01:12:34.667 DEBUG:teuthology.orchestra.run.smithi090:mon.smithi090> sudo journalctl -f -n 0 -u ceph-37f51182-e4c4-11ee-95c9-87774f69a715@mon.smithi090.service 2024-03-18T01:12:34.670 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-18T01:12:34.670 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 37f51182-e4c4-11ee-95c9-87774f69a715 -- ceph mon dump -f json 2024-03-18T01:12:34.944 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-18T01:12:34.991 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-18T01:12:35.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:12:35 smithi029 ceph-mon[29331]: from='client.14190 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi029:172.21.15.29=smithi029;smithi090:172.21.15.90=smithi090", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T01:12:35.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:12:35 smithi029 ceph-mon[29331]: Saving service mon spec with placement smithi029:172.21.15.29=smithi029;smithi090:172.21.15.90=smithi090;count:2 2024-03-18T01:12:35.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:12:35 smithi029 ceph-mon[29331]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-18T01:12:35.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:12:35 smithi029 ceph-mon[29331]: Deploying daemon grafana.smithi029 on smithi029 2024-03-18T01:12:36.300 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:36.299+0000 7ff8fec2e700 1 -- 172.21.15.90:0/2366861467 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff8f80669c0 msgr2=0x7ff8f80fd430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:12:36.300 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:36.299+0000 7ff8fec2e700 1 --2- 172.21.15.90:0/2366861467 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff8f80669c0 0x7ff8f80fd430 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7ff8f40048c0 tx=0x7ff8f4009e30 comp rx=0 tx=0).stop 2024-03-18T01:12:36.300 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:36.299+0000 7ff8fec2e700 1 -- 172.21.15.90:0/2366861467 shutdown_connections 2024-03-18T01:12:36.300 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:36.299+0000 7ff8fec2e700 1 --2- 172.21.15.90:0/2366861467 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff8f80669c0 0x7ff8f80fd430 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:12:36.300 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:36.299+0000 7ff8fec2e700 1 -- 172.21.15.90:0/2366861467 >> 172.21.15.90:0/2366861467 conn(0x7ff8f80f7090 msgr2=0x7ff8f80f94e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:12:36.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:36.299+0000 7ff8fec2e700 1 -- 172.21.15.90:0/2366861467 shutdown_connections 2024-03-18T01:12:36.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:36.299+0000 7ff8fec2e700 1 -- 172.21.15.90:0/2366861467 wait complete. 2024-03-18T01:12:36.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:36.299+0000 7ff8fec2e700 1 Processor -- start 2024-03-18T01:12:36.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:36.299+0000 7ff8fec2e700 1 -- start start 2024-03-18T01:12:36.303 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:36.300+0000 7ff8fec2e700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff8f80669c0 0x7ff8f8113720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:12:36.303 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:36.300+0000 7ff8fec2e700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8f8113c60 con 0x7ff8f80669c0 2024-03-18T01:12:36.303 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:36.300+0000 7ff8fc9ca700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff8f80669c0 0x7ff8f8113720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:12:36.303 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:36.300+0000 7ff8fc9ca700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff8f80669c0 0x7ff8f8113720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.90:36962/0 (socket says 172.21.15.90:36962) 2024-03-18T01:12:36.303 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:36.300+0000 7ff8fc9ca700 1 -- 172.21.15.90:0/3977855606 learned_addr learned my addr 172.21.15.90:0/3977855606 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-18T01:12:36.303 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:36.301+0000 7ff8fc9ca700 1 -- 172.21.15.90:0/3977855606 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8f4005040 con 0x7ff8f80669c0 2024-03-18T01:12:36.303 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:36.301+0000 7ff8fc9ca700 1 --2- 172.21.15.90:0/3977855606 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff8f80669c0 0x7ff8f8113720 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7ff8f4009680 tx=0x7ff8f4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:12:36.303 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:36.301+0000 7ff8ed7fa700 1 -- 172.21.15.90:0/3977855606 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff8f400bed0 con 0x7ff8f80669c0 2024-03-18T01:12:36.303 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:36.301+0000 7ff8ed7fa700 1 -- 172.21.15.90:0/3977855606 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff8f400b5c0 con 0x7ff8f80669c0 2024-03-18T01:12:36.303 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:36.301+0000 7ff8fec2e700 1 -- 172.21.15.90:0/3977855606 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8f8113e60 con 0x7ff8f80669c0 2024-03-18T01:12:36.304 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:36.301+0000 7ff8ed7fa700 1 -- 172.21.15.90:0/3977855606 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff8f401c5b0 con 0x7ff8f80669c0 2024-03-18T01:12:36.304 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:36.301+0000 7ff8fec2e700 1 -- 172.21.15.90:0/3977855606 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8f8114980 con 0x7ff8f80669c0 2024-03-18T01:12:36.304 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:36.302+0000 7ff8ed7fa700 1 -- 172.21.15.90:0/3977855606 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 13) v1 ==== 43424+0+0 (secure 0 0 0) 0x7ff8f401c710 con 0x7ff8f80669c0 2024-03-18T01:12:36.304 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:36.302+0000 7ff8ed7fa700 1 --2- 172.21.15.90:0/3977855606 >> [v2:172.21.15.29:6800/2687730237,v1:172.21.15.29:6801/2687730237] conn(0x7ff8e0036880 0x7ff8e0038d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:12:36.304 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:36.302+0000 7ff8ed7fa700 1 -- 172.21.15.90:0/3977855606 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7ff8f404aae0 con 0x7ff8f80669c0 2024-03-18T01:12:36.304 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:36.302+0000 7ff8effff700 1 --2- 172.21.15.90:0/3977855606 >> [v2:172.21.15.29:6800/2687730237,v1:172.21.15.29:6801/2687730237] conn(0x7ff8e0036880 0x7ff8e0038d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:12:36.304 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:36.302+0000 7ff8fec2e700 1 -- 172.21.15.90:0/3977855606 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8f8100d40 con 0x7ff8f80669c0 2024-03-18T01:12:36.304 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:36.303+0000 7ff8effff700 1 --2- 172.21.15.90:0/3977855606 >> [v2:172.21.15.29:6800/2687730237,v1:172.21.15.29:6801/2687730237] conn(0x7ff8e0036880 0x7ff8e0038d30 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7ff8e8009990 tx=0x7ff8e8006e40 comp rx=0 tx=0).ready entity=mgr.14158 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:12:36.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:36.307+0000 7ff8ed7fa700 1 -- 172.21.15.90:0/3977855606 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175520 (secure 0 0 0) 0x7ff8f4025400 con 0x7ff8f80669c0 2024-03-18T01:12:36.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:36.565+0000 7ff8fec2e700 1 -- 172.21.15.90:0/3977855606 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff8f805f5f0 con 0x7ff8f80669c0 2024-03-18T01:12:36.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:36.565+0000 7ff8ed7fa700 1 -- 172.21.15.90:0/3977855606 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7ff8f805f5f0 con 0x7ff8f80669c0 2024-03-18T01:12:36.566 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-18T01:12:36.566 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":1,"fsid":"37f51182-e4c4-11ee-95c9-87774f69a715","modified":"2024-03-18T01:10:46.941548Z","created":"2024-03-18T01:10:46.941548Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-18T01:12:36.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:36.567+0000 7ff8fec2e700 1 -- 172.21.15.90:0/3977855606 >> [v2:172.21.15.29:6800/2687730237,v1:172.21.15.29:6801/2687730237] conn(0x7ff8e0036880 msgr2=0x7ff8e0038d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:12:36.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:36.567+0000 7ff8fec2e700 1 --2- 172.21.15.90:0/3977855606 >> [v2:172.21.15.29:6800/2687730237,v1:172.21.15.29:6801/2687730237] conn(0x7ff8e0036880 0x7ff8e0038d30 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7ff8e8009990 tx=0x7ff8e8006e40 comp rx=0 tx=0).stop 2024-03-18T01:12:36.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:36.567+0000 7ff8fec2e700 1 -- 172.21.15.90:0/3977855606 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff8f80669c0 msgr2=0x7ff8f8113720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:12:36.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:36.567+0000 7ff8fec2e700 1 --2- 172.21.15.90:0/3977855606 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff8f80669c0 0x7ff8f8113720 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7ff8f4009680 tx=0x7ff8f4009e30 comp rx=0 tx=0).stop 2024-03-18T01:12:36.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:36.567+0000 7ff8fec2e700 1 -- 172.21.15.90:0/3977855606 shutdown_connections 2024-03-18T01:12:36.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:36.567+0000 7ff8fec2e700 1 --2- 172.21.15.90:0/3977855606 >> [v2:172.21.15.29:6800/2687730237,v1:172.21.15.29:6801/2687730237] conn(0x7ff8e0036880 0x7ff8e0038d30 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:12:36.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:36.567+0000 7ff8fec2e700 1 --2- 172.21.15.90:0/3977855606 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff8f80669c0 0x7ff8f8113720 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:12:36.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:36.567+0000 7ff8fec2e700 1 -- 172.21.15.90:0/3977855606 >> 172.21.15.90:0/3977855606 conn(0x7ff8f80f7090 msgr2=0x7ff8f810a250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:12:36.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:36.567+0000 7ff8fec2e700 1 -- 172.21.15.90:0/3977855606 shutdown_connections 2024-03-18T01:12:36.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:36.567+0000 7ff8fec2e700 1 -- 172.21.15.90:0/3977855606 wait complete. 2024-03-18T01:12:36.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:12:36 smithi029 ceph-mon[29331]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T01:12:36.569 INFO:teuthology.orchestra.run.smithi090.stderr:dumped monmap epoch 1 2024-03-18T01:12:37.568 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:12:37 smithi029 ceph-mon[29331]: from='client.? 172.21.15.90:0/3977855606' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-18T01:12:38.143 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-18T01:12:38.143 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 37f51182-e4c4-11ee-95c9-87774f69a715 -- ceph mon dump -f json 2024-03-18T01:12:38.393 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-18T01:12:38.439 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-18T01:12:38.568 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:12:38 smithi029 ceph-mon[29331]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T01:12:39.846 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:39.843+0000 7f464700a700 1 -- 172.21.15.90:0/3398561678 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f46400fca50 msgr2=0x7f46400fce60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:12:39.846 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:39.843+0000 7f464700a700 1 --2- 172.21.15.90:0/3398561678 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f46400fca50 0x7f46400fce60 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f46340048c0 tx=0x7f4634009e30 comp rx=0 tx=0).stop 2024-03-18T01:12:39.846 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:39.843+0000 7f464700a700 1 -- 172.21.15.90:0/3398561678 shutdown_connections 2024-03-18T01:12:39.846 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:39.843+0000 7f464700a700 1 --2- 172.21.15.90:0/3398561678 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f46400fca50 0x7f46400fce60 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:12:39.846 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:39.843+0000 7f464700a700 1 -- 172.21.15.90:0/3398561678 >> 172.21.15.90:0/3398561678 conn(0x7f46400f8190 msgr2=0x7f46400fa5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:12:39.846 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:39.843+0000 7f464700a700 1 -- 172.21.15.90:0/3398561678 shutdown_connections 2024-03-18T01:12:39.846 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:39.843+0000 7f464700a700 1 -- 172.21.15.90:0/3398561678 wait complete. 2024-03-18T01:12:39.846 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:39.844+0000 7f464700a700 1 Processor -- start 2024-03-18T01:12:39.846 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:39.844+0000 7f464700a700 1 -- start start 2024-03-18T01:12:39.846 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:39.844+0000 7f464700a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f46400fca50 0x7f4640071d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:12:39.847 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:39.844+0000 7f464700a700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46400722a0 con 0x7f46400fca50 2024-03-18T01:12:39.847 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:39.844+0000 7f4644da6700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f46400fca50 0x7f4640071d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:12:39.847 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:39.844+0000 7f4644da6700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f46400fca50 0x7f4640071d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.90:57438/0 (socket says 172.21.15.90:57438) 2024-03-18T01:12:39.847 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:39.844+0000 7f4644da6700 1 -- 172.21.15.90:0/497909671 learned_addr learned my addr 172.21.15.90:0/497909671 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-18T01:12:39.847 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:39.845+0000 7f4644da6700 1 -- 172.21.15.90:0/497909671 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4634005040 con 0x7f46400fca50 2024-03-18T01:12:39.847 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:39.845+0000 7f4644da6700 1 --2- 172.21.15.90:0/497909671 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f46400fca50 0x7f4640071d60 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f4634009680 tx=0x7f4634009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:12:39.847 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:39.845+0000 7f463d7fa700 1 -- 172.21.15.90:0/497909671 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f463400b5c0 con 0x7f46400fca50 2024-03-18T01:12:39.847 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:39.845+0000 7f463d7fa700 1 -- 172.21.15.90:0/497909671 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f463400be70 con 0x7f46400fca50 2024-03-18T01:12:39.847 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:39.845+0000 7f464700a700 1 -- 172.21.15.90:0/497909671 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4640070370 con 0x7f46400fca50 2024-03-18T01:12:39.847 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:39.845+0000 7f463d7fa700 1 -- 172.21.15.90:0/497909671 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f46340135a0 con 0x7f46400fca50 2024-03-18T01:12:39.847 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:39.845+0000 7f464700a700 1 -- 172.21.15.90:0/497909671 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4640070860 con 0x7f46400fca50 2024-03-18T01:12:39.848 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:39.846+0000 7f463d7fa700 1 -- 172.21.15.90:0/497909671 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 13) v1 ==== 43424+0+0 (secure 0 0 0) 0x7f463401c430 con 0x7f46400fca50 2024-03-18T01:12:39.848 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:39.846+0000 7f464700a700 1 -- 172.21.15.90:0/497909671 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f464005f5f0 con 0x7f46400fca50 2024-03-18T01:12:39.848 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:39.846+0000 7f463d7fa700 1 --2- 172.21.15.90:0/497909671 >> [v2:172.21.15.29:6800/2687730237,v1:172.21.15.29:6801/2687730237] conn(0x7f4630036880 0x7f4630038d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:12:39.848 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:39.846+0000 7f463d7fa700 1 -- 172.21.15.90:0/497909671 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f46340493c0 con 0x7f46400fca50 2024-03-18T01:12:39.848 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:39.846+0000 7f463ffff700 1 --2- 172.21.15.90:0/497909671 >> [v2:172.21.15.29:6800/2687730237,v1:172.21.15.29:6801/2687730237] conn(0x7f4630036880 0x7f4630038d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:12:39.848 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:39.847+0000 7f463ffff700 1 --2- 172.21.15.90:0/497909671 >> [v2:172.21.15.29:6800/2687730237,v1:172.21.15.29:6801/2687730237] conn(0x7f4630036880 0x7f4630038d30 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f462c009990 tx=0x7f462c006e40 comp rx=0 tx=0).ready entity=mgr.14158 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:12:39.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:39.851+0000 7f463d7fa700 1 -- 172.21.15.90:0/497909671 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175520 (secure 0 0 0) 0x7f4634004d30 con 0x7f46400fca50 2024-03-18T01:12:40.112 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:40.111+0000 7f464700a700 1 -- 172.21.15.90:0/497909671 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f464004bed0 con 0x7f46400fca50 2024-03-18T01:12:40.112 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:40.112+0000 7f463d7fa700 1 -- 172.21.15.90:0/497909671 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7f4634021020 con 0x7f46400fca50 2024-03-18T01:12:40.112 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-18T01:12:40.112 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":1,"fsid":"37f51182-e4c4-11ee-95c9-87774f69a715","modified":"2024-03-18T01:10:46.941548Z","created":"2024-03-18T01:10:46.941548Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-18T01:12:40.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:40.114+0000 7f464700a700 1 -- 172.21.15.90:0/497909671 >> [v2:172.21.15.29:6800/2687730237,v1:172.21.15.29:6801/2687730237] conn(0x7f4630036880 msgr2=0x7f4630038d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:12:40.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:40.114+0000 7f464700a700 1 --2- 172.21.15.90:0/497909671 >> [v2:172.21.15.29:6800/2687730237,v1:172.21.15.29:6801/2687730237] conn(0x7f4630036880 0x7f4630038d30 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f462c009990 tx=0x7f462c006e40 comp rx=0 tx=0).stop 2024-03-18T01:12:40.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:40.114+0000 7f464700a700 1 -- 172.21.15.90:0/497909671 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f46400fca50 msgr2=0x7f4640071d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:12:40.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:40.114+0000 7f464700a700 1 --2- 172.21.15.90:0/497909671 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f46400fca50 0x7f4640071d60 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f4634009680 tx=0x7f4634009e30 comp rx=0 tx=0).stop 2024-03-18T01:12:40.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:40.114+0000 7f464700a700 1 -- 172.21.15.90:0/497909671 shutdown_connections 2024-03-18T01:12:40.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:40.114+0000 7f464700a700 1 --2- 172.21.15.90:0/497909671 >> [v2:172.21.15.29:6800/2687730237,v1:172.21.15.29:6801/2687730237] conn(0x7f4630036880 0x7f4630038d30 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:12:40.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:40.114+0000 7f464700a700 1 --2- 172.21.15.90:0/497909671 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f46400fca50 0x7f4640071d60 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:12:40.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:40.114+0000 7f464700a700 1 -- 172.21.15.90:0/497909671 >> 172.21.15.90:0/497909671 conn(0x7f46400f8190 msgr2=0x7f4640103b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:12:40.115 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:40.114+0000 7f464700a700 1 -- 172.21.15.90:0/497909671 shutdown_connections 2024-03-18T01:12:40.115 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:40.114+0000 7f464700a700 1 -- 172.21.15.90:0/497909671 wait complete. 2024-03-18T01:12:40.115 INFO:teuthology.orchestra.run.smithi090.stderr:dumped monmap epoch 1 2024-03-18T01:12:40.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:12:40 smithi029 ceph-mon[29331]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T01:12:40.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:12:40 smithi029 ceph-mon[29331]: from='client.? 172.21.15.90:0/497909671' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-18T01:12:41.697 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-18T01:12:41.697 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 37f51182-e4c4-11ee-95c9-87774f69a715 -- ceph mon dump -f json 2024-03-18T01:12:41.949 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-18T01:12:41.996 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-18T01:12:42.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:12:42 smithi029 ceph-mon[29331]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T01:12:43.408 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:43.408+0000 7fb549e25700 1 -- 172.21.15.90:0/1971341484 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb544066a40 msgr2=0x7fb5440ff650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:12:43.408 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:43.408+0000 7fb549e25700 1 --2- 172.21.15.90:0/1971341484 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb544066a40 0x7fb5440ff650 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fb5340048c0 tx=0x7fb534009e30 comp rx=0 tx=0).stop 2024-03-18T01:12:43.408 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:43.408+0000 7fb549e25700 1 -- 172.21.15.90:0/1971341484 shutdown_connections 2024-03-18T01:12:43.409 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:43.408+0000 7fb549e25700 1 --2- 172.21.15.90:0/1971341484 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb544066a40 0x7fb5440ff650 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:12:43.409 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:43.408+0000 7fb549e25700 1 -- 172.21.15.90:0/1971341484 >> 172.21.15.90:0/1971341484 conn(0x7fb5440f70c0 msgr2=0x7fb5440f94f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:12:43.409 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:43.408+0000 7fb549e25700 1 -- 172.21.15.90:0/1971341484 shutdown_connections 2024-03-18T01:12:43.409 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:43.408+0000 7fb549e25700 1 -- 172.21.15.90:0/1971341484 wait complete. 2024-03-18T01:12:43.410 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:43.408+0000 7fb549e25700 1 Processor -- start 2024-03-18T01:12:43.410 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:43.409+0000 7fb549e25700 1 -- start start 2024-03-18T01:12:43.410 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:43.409+0000 7fb549e25700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb544066a40 0x7fb5440fcfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:12:43.410 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:43.409+0000 7fb549e25700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5440fd4e0 con 0x7fb544066a40 2024-03-18T01:12:43.410 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:43.409+0000 7fb5437fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb544066a40 0x7fb5440fcfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:12:43.410 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:43.409+0000 7fb5437fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb544066a40 0x7fb5440fcfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.90:57452/0 (socket says 172.21.15.90:57452) 2024-03-18T01:12:43.411 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:43.409+0000 7fb5437fe700 1 -- 172.21.15.90:0/1327578086 learned_addr learned my addr 172.21.15.90:0/1327578086 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-18T01:12:43.411 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:43.409+0000 7fb5437fe700 1 -- 172.21.15.90:0/1327578086 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb534005040 con 0x7fb544066a40 2024-03-18T01:12:43.411 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:43.409+0000 7fb5437fe700 1 --2- 172.21.15.90:0/1327578086 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb544066a40 0x7fb5440fcfa0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fb534009680 tx=0x7fb534009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:12:43.411 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:43.409+0000 7fb52bfff700 1 -- 172.21.15.90:0/1327578086 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb53400bed0 con 0x7fb544066a40 2024-03-18T01:12:43.411 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:43.409+0000 7fb549e25700 1 -- 172.21.15.90:0/1327578086 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5440fb5b0 con 0x7fb544066a40 2024-03-18T01:12:43.411 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:43.409+0000 7fb52bfff700 1 -- 172.21.15.90:0/1327578086 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb53400b5c0 con 0x7fb544066a40 2024-03-18T01:12:43.411 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:43.409+0000 7fb52bfff700 1 -- 172.21.15.90:0/1327578086 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb53401c5b0 con 0x7fb544066a40 2024-03-18T01:12:43.411 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:43.409+0000 7fb549e25700 1 -- 172.21.15.90:0/1327578086 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5440fbaa0 con 0x7fb544066a40 2024-03-18T01:12:43.412 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:43.410+0000 7fb52bfff700 1 -- 172.21.15.90:0/1327578086 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 13) v1 ==== 43424+0+0 (secure 0 0 0) 0x7fb53401c710 con 0x7fb544066a40 2024-03-18T01:12:43.412 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:43.411+0000 7fb52bfff700 1 --2- 172.21.15.90:0/1327578086 >> [v2:172.21.15.29:6800/2687730237,v1:172.21.15.29:6801/2687730237] conn(0x7fb52c036880 0x7fb52c038d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:12:43.412 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:43.411+0000 7fb52bfff700 1 -- 172.21.15.90:0/1327578086 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fb534013a80 con 0x7fb544066a40 2024-03-18T01:12:43.412 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:43.411+0000 7fb549e25700 1 -- 172.21.15.90:0/1327578086 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb54405f5f0 con 0x7fb544066a40 2024-03-18T01:12:43.412 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:43.411+0000 7fb542ffd700 1 --2- 172.21.15.90:0/1327578086 >> [v2:172.21.15.29:6800/2687730237,v1:172.21.15.29:6801/2687730237] conn(0x7fb52c036880 0x7fb52c038d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:12:43.412 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:43.411+0000 7fb542ffd700 1 --2- 172.21.15.90:0/1327578086 >> [v2:172.21.15.29:6800/2687730237,v1:172.21.15.29:6801/2687730237] conn(0x7fb52c036880 0x7fb52c038d30 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fb538009990 tx=0x7fb538006e40 comp rx=0 tx=0).ready entity=mgr.14158 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:12:43.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:43.416+0000 7fb52bfff700 1 -- 172.21.15.90:0/1327578086 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175520 (secure 0 0 0) 0x7fb534025400 con 0x7fb544066a40 2024-03-18T01:12:43.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:43.677+0000 7fb549e25700 1 -- 172.21.15.90:0/1327578086 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb544106430 con 0x7fb544066a40 2024-03-18T01:12:43.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:43.677+0000 7fb52bfff700 1 -- 172.21.15.90:0/1327578086 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7fb534011370 con 0x7fb544066a40 2024-03-18T01:12:43.677 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-18T01:12:43.677 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":1,"fsid":"37f51182-e4c4-11ee-95c9-87774f69a715","modified":"2024-03-18T01:10:46.941548Z","created":"2024-03-18T01:10:46.941548Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-18T01:12:43.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:43.679+0000 7fb549e25700 1 -- 172.21.15.90:0/1327578086 >> [v2:172.21.15.29:6800/2687730237,v1:172.21.15.29:6801/2687730237] conn(0x7fb52c036880 msgr2=0x7fb52c038d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:12:43.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:43.679+0000 7fb549e25700 1 --2- 172.21.15.90:0/1327578086 >> [v2:172.21.15.29:6800/2687730237,v1:172.21.15.29:6801/2687730237] conn(0x7fb52c036880 0x7fb52c038d30 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fb538009990 tx=0x7fb538006e40 comp rx=0 tx=0).stop 2024-03-18T01:12:43.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:43.679+0000 7fb549e25700 1 -- 172.21.15.90:0/1327578086 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb544066a40 msgr2=0x7fb5440fcfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:12:43.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:43.679+0000 7fb549e25700 1 --2- 172.21.15.90:0/1327578086 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb544066a40 0x7fb5440fcfa0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fb534009680 tx=0x7fb534009e30 comp rx=0 tx=0).stop 2024-03-18T01:12:43.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:43.679+0000 7fb549e25700 1 -- 172.21.15.90:0/1327578086 shutdown_connections 2024-03-18T01:12:43.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:43.679+0000 7fb549e25700 1 --2- 172.21.15.90:0/1327578086 >> [v2:172.21.15.29:6800/2687730237,v1:172.21.15.29:6801/2687730237] conn(0x7fb52c036880 0x7fb52c038d30 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:12:43.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:43.679+0000 7fb549e25700 1 --2- 172.21.15.90:0/1327578086 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb544066a40 0x7fb5440fcfa0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:12:43.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:43.679+0000 7fb549e25700 1 -- 172.21.15.90:0/1327578086 >> 172.21.15.90:0/1327578086 conn(0x7fb5440f70c0 msgr2=0x7fb544103b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:12:43.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:43.679+0000 7fb549e25700 1 -- 172.21.15.90:0/1327578086 shutdown_connections 2024-03-18T01:12:43.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:43.679+0000 7fb549e25700 1 -- 172.21.15.90:0/1327578086 wait complete. 2024-03-18T01:12:43.680 INFO:teuthology.orchestra.run.smithi090.stderr:dumped monmap epoch 1 2024-03-18T01:12:44.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:12:44 smithi029 ceph-mon[29331]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T01:12:44.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:12:44 smithi029 ceph-mon[29331]: from='client.? 172.21.15.90:0/1327578086' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-18T01:12:45.254 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-18T01:12:45.255 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 37f51182-e4c4-11ee-95c9-87774f69a715 -- ceph mon dump -f json 2024-03-18T01:12:45.504 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-18T01:12:45.550 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-18T01:12:46.318 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:12:46 smithi029 ceph-mon[29331]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T01:12:46.820 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:46.818+0000 7f648bf73700 1 -- 172.21.15.90:0/3421133719 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f64840fb960 msgr2=0x7f64840fbd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:12:46.820 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:46.818+0000 7f648bf73700 1 --2- 172.21.15.90:0/3421133719 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f64840fb960 0x7f64840fbd70 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f64740048c0 tx=0x7f6474009e30 comp rx=0 tx=0).stop 2024-03-18T01:12:46.820 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:46.818+0000 7f648bf73700 1 -- 172.21.15.90:0/3421133719 shutdown_connections 2024-03-18T01:12:46.820 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:46.818+0000 7f648bf73700 1 --2- 172.21.15.90:0/3421133719 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f64840fb960 0x7f64840fbd70 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:12:46.820 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:46.818+0000 7f648bf73700 1 -- 172.21.15.90:0/3421133719 >> 172.21.15.90:0/3421133719 conn(0x7f64840f70c0 msgr2=0x7f64840f94f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:12:46.821 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:46.818+0000 7f648bf73700 1 -- 172.21.15.90:0/3421133719 shutdown_connections 2024-03-18T01:12:46.821 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:46.819+0000 7f648bf73700 1 -- 172.21.15.90:0/3421133719 wait complete. 2024-03-18T01:12:46.821 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:46.819+0000 7f648bf73700 1 Processor -- start 2024-03-18T01:12:46.821 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:46.819+0000 7f648bf73700 1 -- start start 2024-03-18T01:12:46.821 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:46.819+0000 7f648bf73700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f64840fb960 0x7f64840668c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:12:46.821 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:46.819+0000 7f648bf73700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f648410fd50 con 0x7f64840fb960 2024-03-18T01:12:46.821 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:46.819+0000 7f6489d0f700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f64840fb960 0x7f64840668c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:12:46.821 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:46.819+0000 7f6489d0f700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f64840fb960 0x7f64840668c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.90:49972/0 (socket says 172.21.15.90:49972) 2024-03-18T01:12:46.821 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:46.819+0000 7f6489d0f700 1 -- 172.21.15.90:0/1563475506 learned_addr learned my addr 172.21.15.90:0/1563475506 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-18T01:12:46.821 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:46.820+0000 7f6489d0f700 1 -- 172.21.15.90:0/1563475506 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6474005040 con 0x7f64840fb960 2024-03-18T01:12:46.821 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:46.820+0000 7f6489d0f700 1 --2- 172.21.15.90:0/1563475506 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f64840fb960 0x7f64840668c0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f64740096b0 tx=0x7f6474009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:12:46.821 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:46.820+0000 7f647a7fc700 1 -- 172.21.15.90:0/1563475506 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f647400bed0 con 0x7f64840fb960 2024-03-18T01:12:46.821 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:46.820+0000 7f647a7fc700 1 -- 172.21.15.90:0/1563475506 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f647400b5c0 con 0x7f64840fb960 2024-03-18T01:12:46.821 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:46.820+0000 7f648bf73700 1 -- 172.21.15.90:0/1563475506 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f648410fef0 con 0x7f64840fb960 2024-03-18T01:12:46.821 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:46.820+0000 7f647a7fc700 1 -- 172.21.15.90:0/1563475506 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f647401c5b0 con 0x7f64840fb960 2024-03-18T01:12:46.822 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:46.820+0000 7f648bf73700 1 -- 172.21.15.90:0/1563475506 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6484110380 con 0x7f64840fb960 2024-03-18T01:12:46.822 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:46.821+0000 7f647a7fc700 1 -- 172.21.15.90:0/1563475506 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 13) v1 ==== 43424+0+0 (secure 0 0 0) 0x7f647401c820 con 0x7f64840fb960 2024-03-18T01:12:46.823 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:46.821+0000 7f648bf73700 1 -- 172.21.15.90:0/1563475506 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6484105920 con 0x7f64840fb960 2024-03-18T01:12:46.823 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:46.821+0000 7f647a7fc700 1 --2- 172.21.15.90:0/1563475506 >> [v2:172.21.15.29:6800/2687730237,v1:172.21.15.29:6801/2687730237] conn(0x7f6470036880 0x7f6470038d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:12:46.823 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:46.821+0000 7f647a7fc700 1 -- 172.21.15.90:0/1563475506 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f6474049c10 con 0x7f64840fb960 2024-03-18T01:12:46.823 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:46.821+0000 7f648950e700 1 --2- 172.21.15.90:0/1563475506 >> [v2:172.21.15.29:6800/2687730237,v1:172.21.15.29:6801/2687730237] conn(0x7f6470036880 0x7f6470038d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:12:46.823 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:46.821+0000 7f648950e700 1 --2- 172.21.15.90:0/1563475506 >> [v2:172.21.15.29:6800/2687730237,v1:172.21.15.29:6801/2687730237] conn(0x7f6470036880 0x7f6470038d30 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f6480009990 tx=0x7f6480006e40 comp rx=0 tx=0).ready entity=mgr.14158 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:12:46.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:46.826+0000 7f647a7fc700 1 -- 172.21.15.90:0/1563475506 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175520 (secure 0 0 0) 0x7f6474023300 con 0x7f64840fb960 2024-03-18T01:12:47.081 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:47.081+0000 7f648bf73700 1 -- 172.21.15.90:0/1563475506 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f648405f5f0 con 0x7f64840fb960 2024-03-18T01:12:47.081 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:47.081+0000 7f647a7fc700 1 -- 172.21.15.90:0/1563475506 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7f6474011b10 con 0x7f64840fb960 2024-03-18T01:12:47.082 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-18T01:12:47.082 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":1,"fsid":"37f51182-e4c4-11ee-95c9-87774f69a715","modified":"2024-03-18T01:10:46.941548Z","created":"2024-03-18T01:10:46.941548Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-18T01:12:47.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:47.083+0000 7f648bf73700 1 -- 172.21.15.90:0/1563475506 >> [v2:172.21.15.29:6800/2687730237,v1:172.21.15.29:6801/2687730237] conn(0x7f6470036880 msgr2=0x7f6470038d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:12:47.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:47.083+0000 7f648bf73700 1 --2- 172.21.15.90:0/1563475506 >> [v2:172.21.15.29:6800/2687730237,v1:172.21.15.29:6801/2687730237] conn(0x7f6470036880 0x7f6470038d30 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f6480009990 tx=0x7f6480006e40 comp rx=0 tx=0).stop 2024-03-18T01:12:47.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:47.083+0000 7f648bf73700 1 -- 172.21.15.90:0/1563475506 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f64840fb960 msgr2=0x7f64840668c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:12:47.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:47.083+0000 7f648bf73700 1 --2- 172.21.15.90:0/1563475506 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f64840fb960 0x7f64840668c0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f64740096b0 tx=0x7f6474009e30 comp rx=0 tx=0).stop 2024-03-18T01:12:47.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:47.083+0000 7f648bf73700 1 -- 172.21.15.90:0/1563475506 shutdown_connections 2024-03-18T01:12:47.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:47.083+0000 7f648bf73700 1 --2- 172.21.15.90:0/1563475506 >> [v2:172.21.15.29:6800/2687730237,v1:172.21.15.29:6801/2687730237] conn(0x7f6470036880 0x7f6470038d30 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:12:47.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:47.083+0000 7f648bf73700 1 --2- 172.21.15.90:0/1563475506 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f64840fb960 0x7f64840668c0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:12:47.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:47.083+0000 7f648bf73700 1 -- 172.21.15.90:0/1563475506 >> 172.21.15.90:0/1563475506 conn(0x7f64840f70c0 msgr2=0x7f6484102aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:12:47.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:47.083+0000 7f648bf73700 1 -- 172.21.15.90:0/1563475506 shutdown_connections 2024-03-18T01:12:47.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:47.083+0000 7f648bf73700 1 -- 172.21.15.90:0/1563475506 wait complete. 2024-03-18T01:12:47.084 INFO:teuthology.orchestra.run.smithi090.stderr:dumped monmap epoch 1 2024-03-18T01:12:47.526 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:12:47 smithi029 ceph-mon[29331]: from='client.? 172.21.15.90:0/1563475506' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-18T01:12:48.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:12:48 smithi029 ceph-mon[29331]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T01:12:48.651 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-18T01:12:48.651 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 37f51182-e4c4-11ee-95c9-87774f69a715 -- ceph mon dump -f json 2024-03-18T01:12:48.903 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-18T01:12:48.950 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-18T01:12:50.319 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:12:50 smithi029 ceph-mon[29331]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T01:12:50.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:50.329+0000 7f1c50c83700 1 -- 172.21.15.90:0/621044179 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1c4c0fca50 msgr2=0x7f1c4c0fce60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:12:50.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:50.329+0000 7f1c50c83700 1 --2- 172.21.15.90:0/621044179 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1c4c0fca50 0x7f1c4c0fce60 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f1c44009c60 tx=0x7f1c44004d10 comp rx=0 tx=0).stop 2024-03-18T01:12:50.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:50.329+0000 7f1c50c83700 1 -- 172.21.15.90:0/621044179 shutdown_connections 2024-03-18T01:12:50.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:50.329+0000 7f1c50c83700 1 --2- 172.21.15.90:0/621044179 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1c4c0fca50 0x7f1c4c0fce60 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:12:50.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:50.329+0000 7f1c50c83700 1 -- 172.21.15.90:0/621044179 >> 172.21.15.90:0/621044179 conn(0x7f1c4c0f81d0 msgr2=0x7f1c4c0fa5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:12:50.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:50.329+0000 7f1c50c83700 1 -- 172.21.15.90:0/621044179 shutdown_connections 2024-03-18T01:12:50.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:50.329+0000 7f1c50c83700 1 -- 172.21.15.90:0/621044179 wait complete. 2024-03-18T01:12:50.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:50.330+0000 7f1c50c83700 1 Processor -- start 2024-03-18T01:12:50.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:50.330+0000 7f1c50c83700 1 -- start start 2024-03-18T01:12:50.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:50.330+0000 7f1c50c83700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1c4c0fca50 0x7f1c4c10d080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:12:50.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:50.330+0000 7f1c50c83700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c4c10d5c0 con 0x7f1c4c0fca50 2024-03-18T01:12:50.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:50.330+0000 7f1c4a59c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1c4c0fca50 0x7f1c4c10d080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:12:50.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:50.330+0000 7f1c4a59c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1c4c0fca50 0x7f1c4c10d080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.90:49990/0 (socket says 172.21.15.90:49990) 2024-03-18T01:12:50.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:50.330+0000 7f1c4a59c700 1 -- 172.21.15.90:0/1415162413 learned_addr learned my addr 172.21.15.90:0/1415162413 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-18T01:12:50.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:50.330+0000 7f1c4a59c700 1 -- 172.21.15.90:0/1415162413 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c44009940 con 0x7f1c4c0fca50 2024-03-18T01:12:50.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:50.331+0000 7f1c4a59c700 1 --2- 172.21.15.90:0/1415162413 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1c4c0fca50 0x7f1c4c10d080 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f1c44004e60 tx=0x7f1c44004d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:12:50.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:50.331+0000 7f1c3affd700 1 -- 172.21.15.90:0/1415162413 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1c44012410 con 0x7f1c4c0fca50 2024-03-18T01:12:50.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:50.331+0000 7f1c3affd700 1 -- 172.21.15.90:0/1415162413 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1c44012a40 con 0x7f1c4c0fca50 2024-03-18T01:12:50.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:50.331+0000 7f1c50c83700 1 -- 172.21.15.90:0/1415162413 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c4c10d7c0 con 0x7f1c4c0fca50 2024-03-18T01:12:50.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:50.331+0000 7f1c3affd700 1 -- 172.21.15.90:0/1415162413 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1c4401ba40 con 0x7f1c4c0fca50 2024-03-18T01:12:50.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:50.331+0000 7f1c50c83700 1 -- 172.21.15.90:0/1415162413 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c4c10e330 con 0x7f1c4c0fca50 2024-03-18T01:12:50.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:50.332+0000 7f1c3affd700 1 -- 172.21.15.90:0/1415162413 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 13) v1 ==== 43424+0+0 (secure 0 0 0) 0x7f1c44019040 con 0x7f1c4c0fca50 2024-03-18T01:12:50.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:50.332+0000 7f1c50c83700 1 -- 172.21.15.90:0/1415162413 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c4c102f60 con 0x7f1c4c0fca50 2024-03-18T01:12:50.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:50.332+0000 7f1c3affd700 1 --2- 172.21.15.90:0/1415162413 >> [v2:172.21.15.29:6800/2687730237,v1:172.21.15.29:6801/2687730237] conn(0x7f1c340368d0 0x7f1c34038d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:12:50.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:50.332+0000 7f1c3affd700 1 -- 172.21.15.90:0/1415162413 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f1c44049b50 con 0x7f1c4c0fca50 2024-03-18T01:12:50.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:50.332+0000 7f1c49d9b700 1 --2- 172.21.15.90:0/1415162413 >> [v2:172.21.15.29:6800/2687730237,v1:172.21.15.29:6801/2687730237] conn(0x7f1c340368d0 0x7f1c34038d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:12:50.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:50.332+0000 7f1c49d9b700 1 --2- 172.21.15.90:0/1415162413 >> [v2:172.21.15.29:6800/2687730237,v1:172.21.15.29:6801/2687730237] conn(0x7f1c340368d0 0x7f1c34038d80 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f1c40009990 tx=0x7f1c40006e40 comp rx=0 tx=0).ready entity=mgr.14158 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:12:50.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:50.337+0000 7f1c3affd700 1 -- 172.21.15.90:0/1415162413 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175520 (secure 0 0 0) 0x7f1c440227d0 con 0x7f1c4c0fca50 2024-03-18T01:12:50.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:50.595+0000 7f1c50c83700 1 -- 172.21.15.90:0/1415162413 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1c4c10ed40 con 0x7f1c4c0fca50 2024-03-18T01:12:50.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:50.595+0000 7f1c3affd700 1 -- 172.21.15.90:0/1415162413 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7f1c44017090 con 0x7f1c4c0fca50 2024-03-18T01:12:50.597 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-18T01:12:50.597 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":1,"fsid":"37f51182-e4c4-11ee-95c9-87774f69a715","modified":"2024-03-18T01:10:46.941548Z","created":"2024-03-18T01:10:46.941548Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-18T01:12:50.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:50.598+0000 7f1c50c83700 1 -- 172.21.15.90:0/1415162413 >> [v2:172.21.15.29:6800/2687730237,v1:172.21.15.29:6801/2687730237] conn(0x7f1c340368d0 msgr2=0x7f1c34038d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:12:50.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:50.598+0000 7f1c50c83700 1 --2- 172.21.15.90:0/1415162413 >> [v2:172.21.15.29:6800/2687730237,v1:172.21.15.29:6801/2687730237] conn(0x7f1c340368d0 0x7f1c34038d80 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f1c40009990 tx=0x7f1c40006e40 comp rx=0 tx=0).stop 2024-03-18T01:12:50.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:50.598+0000 7f1c50c83700 1 -- 172.21.15.90:0/1415162413 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1c4c0fca50 msgr2=0x7f1c4c10d080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:12:50.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:50.598+0000 7f1c50c83700 1 --2- 172.21.15.90:0/1415162413 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1c4c0fca50 0x7f1c4c10d080 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f1c44004e60 tx=0x7f1c44004d10 comp rx=0 tx=0).stop 2024-03-18T01:12:50.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:50.598+0000 7f1c50c83700 1 -- 172.21.15.90:0/1415162413 shutdown_connections 2024-03-18T01:12:50.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:50.598+0000 7f1c50c83700 1 --2- 172.21.15.90:0/1415162413 >> [v2:172.21.15.29:6800/2687730237,v1:172.21.15.29:6801/2687730237] conn(0x7f1c340368d0 0x7f1c34038d80 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:12:50.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:50.598+0000 7f1c50c83700 1 --2- 172.21.15.90:0/1415162413 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1c4c0fca50 0x7f1c4c10d080 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:12:50.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:50.598+0000 7f1c50c83700 1 -- 172.21.15.90:0/1415162413 >> 172.21.15.90:0/1415162413 conn(0x7f1c4c0f81d0 msgr2=0x7f1c4c103b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:12:50.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:50.598+0000 7f1c50c83700 1 -- 172.21.15.90:0/1415162413 shutdown_connections 2024-03-18T01:12:50.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:50.598+0000 7f1c50c83700 1 -- 172.21.15.90:0/1415162413 wait complete. 2024-03-18T01:12:50.600 INFO:teuthology.orchestra.run.smithi090.stderr:dumped monmap epoch 1 2024-03-18T01:12:51.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:12:51 smithi029 ceph-mon[29331]: from='client.? 172.21.15.90:0/1415162413' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-18T01:12:51.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:12:51 smithi029 ceph-mon[29331]: from='mgr.14158 172.21.15.29:0/4059439045' entity='mgr.smithi029.qviwhl' 2024-03-18T01:12:51.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:12:51 smithi029 ceph-mon[29331]: from='mgr.14158 172.21.15.29:0/4059439045' entity='mgr.smithi029.qviwhl' 2024-03-18T01:12:51.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:12:51 smithi029 ceph-mon[29331]: from='mgr.14158 172.21.15.29:0/4059439045' entity='mgr.smithi029.qviwhl' 2024-03-18T01:12:51.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:12:51 smithi029 ceph-mon[29331]: from='mgr.14158 172.21.15.29:0/4059439045' entity='mgr.smithi029.qviwhl' 2024-03-18T01:12:51.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:12:51 smithi029 ceph-mon[29331]: from='mgr.14158 172.21.15.29:0/4059439045' entity='mgr.smithi029.qviwhl' 2024-03-18T01:12:51.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:12:51 smithi029 ceph-mon[29331]: from='mgr.14158 172.21.15.29:0/4059439045' entity='mgr.smithi029.qviwhl' 2024-03-18T01:12:52.170 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-18T01:12:52.171 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 37f51182-e4c4-11ee-95c9-87774f69a715 -- ceph mon dump -f json 2024-03-18T01:12:52.417 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-18T01:12:52.465 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-18T01:12:52.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:12:52 smithi029 ceph-mon[29331]: Deploying daemon prometheus.smithi029 on smithi029 2024-03-18T01:12:52.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:12:52 smithi029 ceph-mon[29331]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T01:12:53.798 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:53.796+0000 7f28cb7dc700 1 -- 172.21.15.90:0/1955088641 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f28c40fb940 msgr2=0x7f28c40fbd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:12:53.798 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:53.796+0000 7f28cb7dc700 1 --2- 172.21.15.90:0/1955088641 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f28c40fb940 0x7f28c40fbd50 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f28c00048c0 tx=0x7f28c0009e30 comp rx=0 tx=0).stop 2024-03-18T01:12:53.798 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:53.796+0000 7f28cb7dc700 1 -- 172.21.15.90:0/1955088641 shutdown_connections 2024-03-18T01:12:53.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:53.796+0000 7f28cb7dc700 1 --2- 172.21.15.90:0/1955088641 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f28c40fb940 0x7f28c40fbd50 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:12:53.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:53.796+0000 7f28cb7dc700 1 -- 172.21.15.90:0/1955088641 >> 172.21.15.90:0/1955088641 conn(0x7f28c40f70c0 msgr2=0x7f28c40f94d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:12:53.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:53.797+0000 7f28cb7dc700 1 -- 172.21.15.90:0/1955088641 shutdown_connections 2024-03-18T01:12:53.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:53.797+0000 7f28cb7dc700 1 -- 172.21.15.90:0/1955088641 wait complete. 2024-03-18T01:12:53.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:53.797+0000 7f28cb7dc700 1 Processor -- start 2024-03-18T01:12:53.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:53.797+0000 7f28cb7dc700 1 -- start start 2024-03-18T01:12:53.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:53.797+0000 7f28cb7dc700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f28c40fb940 0x7f28c410d0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:12:53.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:53.797+0000 7f28cb7dc700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28c410d5e0 con 0x7f28c40fb940 2024-03-18T01:12:53.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:53.797+0000 7f28c9578700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f28c40fb940 0x7f28c410d0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:12:53.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:53.797+0000 7f28c9578700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f28c40fb940 0x7f28c410d0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.90:50014/0 (socket says 172.21.15.90:50014) 2024-03-18T01:12:53.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:53.797+0000 7f28c9578700 1 -- 172.21.15.90:0/3655461166 learned_addr learned my addr 172.21.15.90:0/3655461166 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-18T01:12:53.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:53.798+0000 7f28c9578700 1 -- 172.21.15.90:0/3655461166 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28c0005040 con 0x7f28c40fb940 2024-03-18T01:12:53.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:53.798+0000 7f28c9578700 1 --2- 172.21.15.90:0/3655461166 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f28c40fb940 0x7f28c410d0a0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f28c00096b0 tx=0x7f28c0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:12:53.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:53.798+0000 7f28b9ffb700 1 -- 172.21.15.90:0/3655461166 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f28c000bed0 con 0x7f28c40fb940 2024-03-18T01:12:53.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:53.798+0000 7f28b9ffb700 1 -- 172.21.15.90:0/3655461166 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f28c000b5c0 con 0x7f28c40fb940 2024-03-18T01:12:53.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:53.798+0000 7f28cb7dc700 1 -- 172.21.15.90:0/3655461166 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28c410d7e0 con 0x7f28c40fb940 2024-03-18T01:12:53.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:53.798+0000 7f28b9ffb700 1 -- 172.21.15.90:0/3655461166 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f28c001c5b0 con 0x7f28c40fb940 2024-03-18T01:12:53.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:53.798+0000 7f28cb7dc700 1 -- 172.21.15.90:0/3655461166 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28c410e350 con 0x7f28c40fb940 2024-03-18T01:12:53.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:53.799+0000 7f28b9ffb700 1 -- 172.21.15.90:0/3655461166 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 13) v1 ==== 43424+0+0 (secure 0 0 0) 0x7f28c001c710 con 0x7f28c40fb940 2024-03-18T01:12:53.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:53.799+0000 7f28b9ffb700 1 --2- 172.21.15.90:0/3655461166 >> [v2:172.21.15.29:6800/2687730237,v1:172.21.15.29:6801/2687730237] conn(0x7f28b0036920 0x7f28b0038dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:12:53.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:53.799+0000 7f28b9ffb700 1 -- 172.21.15.90:0/3655461166 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f28c0049a70 con 0x7f28c40fb940 2024-03-18T01:12:53.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:53.799+0000 7f28cb7dc700 1 -- 172.21.15.90:0/3655461166 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28c4101e50 con 0x7f28c40fb940 2024-03-18T01:12:53.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:53.799+0000 7f28c8d77700 1 --2- 172.21.15.90:0/3655461166 >> [v2:172.21.15.29:6800/2687730237,v1:172.21.15.29:6801/2687730237] conn(0x7f28b0036920 0x7f28b0038dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:12:53.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:53.800+0000 7f28c8d77700 1 --2- 172.21.15.90:0/3655461166 >> [v2:172.21.15.29:6800/2687730237,v1:172.21.15.29:6801/2687730237] conn(0x7f28b0036920 0x7f28b0038dd0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f28b4009990 tx=0x7f28b4006e40 comp rx=0 tx=0).ready entity=mgr.14158 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:12:53.804 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:53.804+0000 7f28b9ffb700 1 -- 172.21.15.90:0/3655461166 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175520 (secure 0 0 0) 0x7f28c0025400 con 0x7f28c40fb940 2024-03-18T01:12:54.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:54.057+0000 7f28cb7dc700 1 -- 172.21.15.90:0/3655461166 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f28c404bed0 con 0x7f28c40fb940 2024-03-18T01:12:54.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:54.057+0000 7f28b9ffb700 1 -- 172.21.15.90:0/3655461166 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7f28c0034090 con 0x7f28c40fb940 2024-03-18T01:12:54.058 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-18T01:12:54.058 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":1,"fsid":"37f51182-e4c4-11ee-95c9-87774f69a715","modified":"2024-03-18T01:10:46.941548Z","created":"2024-03-18T01:10:46.941548Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-18T01:12:54.060 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:54.059+0000 7f28cb7dc700 1 -- 172.21.15.90:0/3655461166 >> [v2:172.21.15.29:6800/2687730237,v1:172.21.15.29:6801/2687730237] conn(0x7f28b0036920 msgr2=0x7f28b0038dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:12:54.060 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:54.059+0000 7f28cb7dc700 1 --2- 172.21.15.90:0/3655461166 >> [v2:172.21.15.29:6800/2687730237,v1:172.21.15.29:6801/2687730237] conn(0x7f28b0036920 0x7f28b0038dd0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f28b4009990 tx=0x7f28b4006e40 comp rx=0 tx=0).stop 2024-03-18T01:12:54.060 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:54.059+0000 7f28cb7dc700 1 -- 172.21.15.90:0/3655461166 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f28c40fb940 msgr2=0x7f28c410d0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:12:54.060 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:54.059+0000 7f28cb7dc700 1 --2- 172.21.15.90:0/3655461166 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f28c40fb940 0x7f28c410d0a0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f28c00096b0 tx=0x7f28c0009e30 comp rx=0 tx=0).stop 2024-03-18T01:12:54.060 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:54.059+0000 7f28cb7dc700 1 -- 172.21.15.90:0/3655461166 shutdown_connections 2024-03-18T01:12:54.060 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:54.059+0000 7f28cb7dc700 1 --2- 172.21.15.90:0/3655461166 >> [v2:172.21.15.29:6800/2687730237,v1:172.21.15.29:6801/2687730237] conn(0x7f28b0036920 0x7f28b0038dd0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:12:54.060 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:54.059+0000 7f28cb7dc700 1 --2- 172.21.15.90:0/3655461166 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f28c40fb940 0x7f28c410d0a0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:12:54.060 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:54.059+0000 7f28cb7dc700 1 -- 172.21.15.90:0/3655461166 >> 172.21.15.90:0/3655461166 conn(0x7f28c40f70c0 msgr2=0x7f28c4065ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:12:54.060 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:54.059+0000 7f28cb7dc700 1 -- 172.21.15.90:0/3655461166 shutdown_connections 2024-03-18T01:12:54.060 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:54.059+0000 7f28cb7dc700 1 -- 172.21.15.90:0/3655461166 wait complete. 2024-03-18T01:12:54.061 INFO:teuthology.orchestra.run.smithi090.stderr:dumped monmap epoch 1 2024-03-18T01:12:54.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:12:54 smithi029 ceph-mon[29331]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T01:12:54.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:12:54 smithi029 ceph-mon[29331]: from='client.? 172.21.15.90:0/3655461166' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-18T01:12:54.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:12:54 smithi029 ceph-mon[29331]: from='mgr.14158 172.21.15.29:0/4059439045' entity='mgr.smithi029.qviwhl' 2024-03-18T01:12:55.643 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-18T01:12:55.644 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 37f51182-e4c4-11ee-95c9-87774f69a715 -- ceph mon dump -f json 2024-03-18T01:12:55.893 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-18T01:12:55.941 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-18T01:12:56.319 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:12:56 smithi029 ceph-mon[29331]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T01:12:57.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:57.401+0000 7fd7526dc700 1 -- 172.21.15.90:0/3635873827 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd74c0fb950 msgr2=0x7fd74c0fbd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:12:57.403 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:57.401+0000 7fd7526dc700 1 --2- 172.21.15.90:0/3635873827 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd74c0fb950 0x7fd74c0fbd60 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fd73c0048c0 tx=0x7fd73c009e30 comp rx=0 tx=0).stop 2024-03-18T01:12:57.403 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:57.401+0000 7fd7526dc700 1 -- 172.21.15.90:0/3635873827 shutdown_connections 2024-03-18T01:12:57.403 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:57.401+0000 7fd7526dc700 1 --2- 172.21.15.90:0/3635873827 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd74c0fb950 0x7fd74c0fbd60 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:12:57.403 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:57.401+0000 7fd7526dc700 1 -- 172.21.15.90:0/3635873827 >> 172.21.15.90:0/3635873827 conn(0x7fd74c0f7090 msgr2=0x7fd74c0f94e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:12:57.403 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:57.401+0000 7fd7526dc700 1 -- 172.21.15.90:0/3635873827 shutdown_connections 2024-03-18T01:12:57.403 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:57.401+0000 7fd7526dc700 1 -- 172.21.15.90:0/3635873827 wait complete. 2024-03-18T01:12:57.403 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:57.401+0000 7fd7526dc700 1 Processor -- start 2024-03-18T01:12:57.403 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:57.401+0000 7fd7526dc700 1 -- start start 2024-03-18T01:12:57.403 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:57.401+0000 7fd7526dc700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd74c0fb950 0x7fd74c066840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:12:57.403 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:57.401+0000 7fd7526dc700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd74c10fda0 con 0x7fd74c0fb950 2024-03-18T01:12:57.403 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:57.402+0000 7fd74bfff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd74c0fb950 0x7fd74c066840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:12:57.404 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:57.402+0000 7fd74bfff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd74c0fb950 0x7fd74c066840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.90:58138/0 (socket says 172.21.15.90:58138) 2024-03-18T01:12:57.404 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:57.402+0000 7fd74bfff700 1 -- 172.21.15.90:0/2623336486 learned_addr learned my addr 172.21.15.90:0/2623336486 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-18T01:12:57.404 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:57.402+0000 7fd74bfff700 1 -- 172.21.15.90:0/2623336486 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd73c005040 con 0x7fd74c0fb950 2024-03-18T01:12:57.404 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:57.402+0000 7fd74bfff700 1 --2- 172.21.15.90:0/2623336486 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd74c0fb950 0x7fd74c066840 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fd73c0096b0 tx=0x7fd73c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:12:57.404 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:57.402+0000 7fd748ff9700 1 -- 172.21.15.90:0/2623336486 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd73c00bed0 con 0x7fd74c0fb950 2024-03-18T01:12:57.404 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:57.402+0000 7fd748ff9700 1 -- 172.21.15.90:0/2623336486 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd73c00b5c0 con 0x7fd74c0fb950 2024-03-18T01:12:57.404 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:57.402+0000 7fd7526dc700 1 -- 172.21.15.90:0/2623336486 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd74c10fee0 con 0x7fd74c0fb950 2024-03-18T01:12:57.404 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:57.402+0000 7fd748ff9700 1 -- 172.21.15.90:0/2623336486 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd73c01c5b0 con 0x7fd74c0fb950 2024-03-18T01:12:57.404 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:57.402+0000 7fd7526dc700 1 -- 172.21.15.90:0/2623336486 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd74c1103d0 con 0x7fd74c0fb950 2024-03-18T01:12:57.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:57.403+0000 7fd748ff9700 1 -- 172.21.15.90:0/2623336486 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 13) v1 ==== 43424+0+0 (secure 0 0 0) 0x7fd73c01c820 con 0x7fd74c0fb950 2024-03-18T01:12:57.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:57.403+0000 7fd7526dc700 1 -- 172.21.15.90:0/2623336486 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd74c105910 con 0x7fd74c0fb950 2024-03-18T01:12:57.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:57.403+0000 7fd748ff9700 1 --2- 172.21.15.90:0/2623336486 >> [v2:172.21.15.29:6800/2687730237,v1:172.21.15.29:6801/2687730237] conn(0x7fd734036880 0x7fd734038d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:12:57.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:57.403+0000 7fd748ff9700 1 -- 172.21.15.90:0/2623336486 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fd73c049b70 con 0x7fd74c0fb950 2024-03-18T01:12:57.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:57.404+0000 7fd74b7fe700 1 --2- 172.21.15.90:0/2623336486 >> [v2:172.21.15.29:6800/2687730237,v1:172.21.15.29:6801/2687730237] conn(0x7fd734036880 0x7fd734038d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:12:57.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:57.404+0000 7fd74b7fe700 1 --2- 172.21.15.90:0/2623336486 >> [v2:172.21.15.29:6800/2687730237,v1:172.21.15.29:6801/2687730237] conn(0x7fd734036880 0x7fd734038d30 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fd740009990 tx=0x7fd740006e40 comp rx=0 tx=0).ready entity=mgr.14158 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:12:57.409 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:57.409+0000 7fd748ff9700 1 -- 172.21.15.90:0/2623336486 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175520 (secure 0 0 0) 0x7fd73c0119c0 con 0x7fd74c0fb950 2024-03-18T01:12:57.665 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:57.664+0000 7fd7526dc700 1 -- 172.21.15.90:0/2623336486 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd74c05f5f0 con 0x7fd74c0fb950 2024-03-18T01:12:57.665 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:57.665+0000 7fd748ff9700 1 -- 172.21.15.90:0/2623336486 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7fd73c0117e0 con 0x7fd74c0fb950 2024-03-18T01:12:57.665 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-18T01:12:57.665 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":1,"fsid":"37f51182-e4c4-11ee-95c9-87774f69a715","modified":"2024-03-18T01:10:46.941548Z","created":"2024-03-18T01:10:46.941548Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-18T01:12:57.667 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:57.667+0000 7fd7526dc700 1 -- 172.21.15.90:0/2623336486 >> [v2:172.21.15.29:6800/2687730237,v1:172.21.15.29:6801/2687730237] conn(0x7fd734036880 msgr2=0x7fd734038d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:12:57.667 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:57.667+0000 7fd7526dc700 1 --2- 172.21.15.90:0/2623336486 >> [v2:172.21.15.29:6800/2687730237,v1:172.21.15.29:6801/2687730237] conn(0x7fd734036880 0x7fd734038d30 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fd740009990 tx=0x7fd740006e40 comp rx=0 tx=0).stop 2024-03-18T01:12:57.667 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:57.667+0000 7fd7526dc700 1 -- 172.21.15.90:0/2623336486 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd74c0fb950 msgr2=0x7fd74c066840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:12:57.667 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:57.667+0000 7fd7526dc700 1 --2- 172.21.15.90:0/2623336486 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd74c0fb950 0x7fd74c066840 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fd73c0096b0 tx=0x7fd73c009e30 comp rx=0 tx=0).stop 2024-03-18T01:12:57.668 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:57.667+0000 7fd7526dc700 1 -- 172.21.15.90:0/2623336486 shutdown_connections 2024-03-18T01:12:57.668 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:57.667+0000 7fd7526dc700 1 --2- 172.21.15.90:0/2623336486 >> [v2:172.21.15.29:6800/2687730237,v1:172.21.15.29:6801/2687730237] conn(0x7fd734036880 0x7fd734038d30 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:12:57.668 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:57.667+0000 7fd7526dc700 1 --2- 172.21.15.90:0/2623336486 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd74c0fb950 0x7fd74c066840 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:12:57.668 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:57.667+0000 7fd7526dc700 1 -- 172.21.15.90:0/2623336486 >> 172.21.15.90:0/2623336486 conn(0x7fd74c0f7090 msgr2=0x7fd74c102a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:12:57.668 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:57.667+0000 7fd7526dc700 1 -- 172.21.15.90:0/2623336486 shutdown_connections 2024-03-18T01:12:57.668 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:12:57.667+0000 7fd7526dc700 1 -- 172.21.15.90:0/2623336486 wait complete. 2024-03-18T01:12:57.668 INFO:teuthology.orchestra.run.smithi090.stderr:dumped monmap epoch 1 2024-03-18T01:12:58.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:12:58 smithi029 ceph-mon[29331]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T01:12:58.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:12:58 smithi029 ceph-mon[29331]: from='client.? 172.21.15.90:0/2623336486' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-18T01:12:59.221 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-18T01:12:59.222 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 37f51182-e4c4-11ee-95c9-87774f69a715 -- ceph mon dump -f json 2024-03-18T01:12:59.471 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-18T01:12:59.519 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-18T01:13:00.539 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:13:00 smithi029 ceph-mon[29331]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T01:13:00.813 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:00.812+0000 7f08c38e6700 1 -- 172.21.15.90:0/3122670577 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f08bc0fe0b0 msgr2=0x7f08bc0fe4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:13:00.814 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:00.812+0000 7f08c38e6700 1 --2- 172.21.15.90:0/3122670577 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f08bc0fe0b0 0x7f08bc0fe4c0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f08b4009c60 tx=0x7f08b4004d10 comp rx=0 tx=0).stop 2024-03-18T01:13:00.814 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:00.812+0000 7f08c38e6700 1 -- 172.21.15.90:0/3122670577 shutdown_connections 2024-03-18T01:13:00.814 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:00.812+0000 7f08c38e6700 1 --2- 172.21.15.90:0/3122670577 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f08bc0fe0b0 0x7f08bc0fe4c0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:13:00.814 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:00.812+0000 7f08c38e6700 1 -- 172.21.15.90:0/3122670577 >> 172.21.15.90:0/3122670577 conn(0x7f08bc0f6850 msgr2=0x7f08bc0f8c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:13:00.814 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:00.812+0000 7f08c38e6700 1 -- 172.21.15.90:0/3122670577 shutdown_connections 2024-03-18T01:13:00.814 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:00.812+0000 7f08c38e6700 1 -- 172.21.15.90:0/3122670577 wait complete. 2024-03-18T01:13:00.815 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:00.812+0000 7f08c38e6700 1 Processor -- start 2024-03-18T01:13:00.815 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:00.812+0000 7f08c38e6700 1 -- start start 2024-03-18T01:13:00.815 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:00.812+0000 7f08c38e6700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f08bc0fe0b0 0x7f08bc10d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:13:00.815 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:00.812+0000 7f08c38e6700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08bc10d630 con 0x7f08bc0fe0b0 2024-03-18T01:13:00.815 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:00.813+0000 7f08c1682700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f08bc0fe0b0 0x7f08bc10d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:13:00.816 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:00.813+0000 7f08c1682700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f08bc0fe0b0 0x7f08bc10d0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.90:58154/0 (socket says 172.21.15.90:58154) 2024-03-18T01:13:00.816 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:00.813+0000 7f08c1682700 1 -- 172.21.15.90:0/758813698 learned_addr learned my addr 172.21.15.90:0/758813698 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-18T01:13:00.816 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:00.813+0000 7f08c1682700 1 -- 172.21.15.90:0/758813698 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08b4009940 con 0x7f08bc0fe0b0 2024-03-18T01:13:00.816 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:00.813+0000 7f08c1682700 1 --2- 172.21.15.90:0/758813698 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f08bc0fe0b0 0x7f08bc10d0f0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f08b4009500 tx=0x7f08b4004d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:13:00.816 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:00.813+0000 7f08b1ffb700 1 -- 172.21.15.90:0/758813698 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f08b4007c80 con 0x7f08bc0fe0b0 2024-03-18T01:13:00.816 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:00.813+0000 7f08c38e6700 1 -- 172.21.15.90:0/758813698 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08bc10d830 con 0x7f08bc0fe0b0 2024-03-18T01:13:00.817 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:00.813+0000 7f08b1ffb700 1 -- 172.21.15.90:0/758813698 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f08b401d910 con 0x7f08bc0fe0b0 2024-03-18T01:13:00.817 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:00.813+0000 7f08b1ffb700 1 -- 172.21.15.90:0/758813698 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f08b40129d0 con 0x7f08bc0fe0b0 2024-03-18T01:13:00.817 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:00.814+0000 7f08c38e6700 1 -- 172.21.15.90:0/758813698 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08bc10e3a0 con 0x7f08bc0fe0b0 2024-03-18T01:13:00.817 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:00.814+0000 7f08c38e6700 1 -- 172.21.15.90:0/758813698 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08bc102f80 con 0x7f08bc0fe0b0 2024-03-18T01:13:00.818 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:00.815+0000 7f08b1ffb700 1 -- 172.21.15.90:0/758813698 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 13) v1 ==== 43424+0+0 (secure 0 0 0) 0x7f08b4019040 con 0x7f08bc0fe0b0 2024-03-18T01:13:00.818 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:00.815+0000 7f08b1ffb700 1 --2- 172.21.15.90:0/758813698 >> [v2:172.21.15.29:6800/2687730237,v1:172.21.15.29:6801/2687730237] conn(0x7f08a8036880 0x7f08a8038d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:13:00.818 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:00.815+0000 7f08b1ffb700 1 -- 172.21.15.90:0/758813698 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f08b404a320 con 0x7f08bc0fe0b0 2024-03-18T01:13:00.818 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:00.815+0000 7f08c0e81700 1 --2- 172.21.15.90:0/758813698 >> [v2:172.21.15.29:6800/2687730237,v1:172.21.15.29:6801/2687730237] conn(0x7f08a8036880 0x7f08a8038d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:13:00.818 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:00.815+0000 7f08c0e81700 1 --2- 172.21.15.90:0/758813698 >> [v2:172.21.15.29:6800/2687730237,v1:172.21.15.29:6801/2687730237] conn(0x7f08a8036880 0x7f08a8038d30 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f08ac009990 tx=0x7f08ac006e40 comp rx=0 tx=0).ready entity=mgr.14158 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:13:00.821 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:00.820+0000 7f08b1ffb700 1 -- 172.21.15.90:0/758813698 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175520 (secure 0 0 0) 0x7f08b402eb40 con 0x7f08bc0fe0b0 2024-03-18T01:13:01.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:01.075+0000 7f08c38e6700 1 -- 172.21.15.90:0/758813698 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f08bc10ed00 con 0x7f08bc0fe0b0 2024-03-18T01:13:01.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:01.076+0000 7f08b1ffb700 1 -- 172.21.15.90:0/758813698 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7f08b4017020 con 0x7f08bc0fe0b0 2024-03-18T01:13:01.077 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-18T01:13:01.078 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":1,"fsid":"37f51182-e4c4-11ee-95c9-87774f69a715","modified":"2024-03-18T01:10:46.941548Z","created":"2024-03-18T01:10:46.941548Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-18T01:13:01.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:01.078+0000 7f08c38e6700 1 -- 172.21.15.90:0/758813698 >> [v2:172.21.15.29:6800/2687730237,v1:172.21.15.29:6801/2687730237] conn(0x7f08a8036880 msgr2=0x7f08a8038d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:13:01.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:01.078+0000 7f08c38e6700 1 --2- 172.21.15.90:0/758813698 >> [v2:172.21.15.29:6800/2687730237,v1:172.21.15.29:6801/2687730237] conn(0x7f08a8036880 0x7f08a8038d30 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f08ac009990 tx=0x7f08ac006e40 comp rx=0 tx=0).stop 2024-03-18T01:13:01.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:01.078+0000 7f08c38e6700 1 -- 172.21.15.90:0/758813698 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f08bc0fe0b0 msgr2=0x7f08bc10d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:13:01.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:01.078+0000 7f08c38e6700 1 --2- 172.21.15.90:0/758813698 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f08bc0fe0b0 0x7f08bc10d0f0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f08b4009500 tx=0x7f08b4004d10 comp rx=0 tx=0).stop 2024-03-18T01:13:01.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:01.078+0000 7f08c38e6700 1 -- 172.21.15.90:0/758813698 shutdown_connections 2024-03-18T01:13:01.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:01.078+0000 7f08c38e6700 1 --2- 172.21.15.90:0/758813698 >> [v2:172.21.15.29:6800/2687730237,v1:172.21.15.29:6801/2687730237] conn(0x7f08a8036880 0x7f08a8038d30 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:13:01.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:01.078+0000 7f08c38e6700 1 --2- 172.21.15.90:0/758813698 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f08bc0fe0b0 0x7f08bc10d0f0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:13:01.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:01.078+0000 7f08c38e6700 1 -- 172.21.15.90:0/758813698 >> 172.21.15.90:0/758813698 conn(0x7f08bc0f6850 msgr2=0x7f08bc103bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:13:01.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:01.078+0000 7f08c38e6700 1 -- 172.21.15.90:0/758813698 shutdown_connections 2024-03-18T01:13:01.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:01.078+0000 7f08c38e6700 1 -- 172.21.15.90:0/758813698 wait complete. 2024-03-18T01:13:01.080 INFO:teuthology.orchestra.run.smithi090.stderr:dumped monmap epoch 1 2024-03-18T01:13:01.546 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:13:01 smithi029 ceph-mon[29331]: from='client.? 172.21.15.90:0/758813698' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-18T01:13:02.510 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:13:02 smithi029 ceph-mon[29331]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T01:13:02.656 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-18T01:13:02.656 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 37f51182-e4c4-11ee-95c9-87774f69a715 -- ceph mon dump -f json 2024-03-18T01:13:02.903 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-18T01:13:02.950 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-18T01:13:04.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:04.552+0000 7f6188ab3700 1 -- 172.21.15.90:0/4086414138 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f61840fb940 msgr2=0x7f61840fbd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:13:04.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:04.552+0000 7f6188ab3700 1 --2- 172.21.15.90:0/4086414138 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f61840fb940 0x7f61840fbd50 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f61780048c0 tx=0x7f6178009e30 comp rx=0 tx=0).stop 2024-03-18T01:13:04.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:04.552+0000 7f6188ab3700 1 -- 172.21.15.90:0/4086414138 shutdown_connections 2024-03-18T01:13:04.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:04.552+0000 7f6188ab3700 1 --2- 172.21.15.90:0/4086414138 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f61840fb940 0x7f61840fbd50 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:13:04.556 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:04.552+0000 7f6188ab3700 1 -- 172.21.15.90:0/4086414138 >> 172.21.15.90:0/4086414138 conn(0x7f61840f70c0 msgr2=0x7f61840f94d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:13:04.556 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:04.552+0000 7f6188ab3700 1 -- 172.21.15.90:0/4086414138 shutdown_connections 2024-03-18T01:13:04.556 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:04.552+0000 7f6188ab3700 1 -- 172.21.15.90:0/4086414138 wait complete. 2024-03-18T01:13:04.556 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:04.553+0000 7f6188ab3700 1 Processor -- start 2024-03-18T01:13:04.556 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:04.553+0000 7f6188ab3700 1 -- start start 2024-03-18T01:13:04.556 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:04.553+0000 7f6188ab3700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f61840fb940 0x7f61840f9000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:13:04.556 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:04.553+0000 7f6188ab3700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61840f9540 con 0x7f61840fb940 2024-03-18T01:13:04.556 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:04.553+0000 7f618259c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f61840fb940 0x7f61840f9000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:13:04.556 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:04.553+0000 7f618259c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f61840fb940 0x7f61840f9000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.90:58166/0 (socket says 172.21.15.90:58166) 2024-03-18T01:13:04.556 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:04.553+0000 7f618259c700 1 -- 172.21.15.90:0/3886463845 learned_addr learned my addr 172.21.15.90:0/3886463845 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-18T01:13:04.556 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:04.553+0000 7f618259c700 1 -- 172.21.15.90:0/3886463845 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6178005040 con 0x7f61840fb940 2024-03-18T01:13:04.556 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:04.554+0000 7f618259c700 1 --2- 172.21.15.90:0/3886463845 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f61840fb940 0x7f61840f9000 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f61780096b0 tx=0x7f6178009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:13:04.556 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:04.554+0000 7f6172ffd700 1 -- 172.21.15.90:0/3886463845 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f617800bed0 con 0x7f61840fb940 2024-03-18T01:13:04.556 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:04.554+0000 7f6172ffd700 1 -- 172.21.15.90:0/3886463845 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f617800b5c0 con 0x7f61840fb940 2024-03-18T01:13:04.556 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:04.554+0000 7f6188ab3700 1 -- 172.21.15.90:0/3886463845 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61840f7610 con 0x7f61840fb940 2024-03-18T01:13:04.557 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:04.554+0000 7f6172ffd700 1 -- 172.21.15.90:0/3886463845 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f617801c5b0 con 0x7f61840fb940 2024-03-18T01:13:04.557 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:04.554+0000 7f6188ab3700 1 -- 172.21.15.90:0/3886463845 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61840f7b00 con 0x7f61840fb940 2024-03-18T01:13:04.557 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:04.555+0000 7f6172ffd700 1 -- 172.21.15.90:0/3886463845 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 14) v1 ==== 43438+0+0 (secure 0 0 0) 0x7f617801c820 con 0x7f61840fb940 2024-03-18T01:13:04.558 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:04.555+0000 7f6188ab3700 1 -- 172.21.15.90:0/3886463845 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f618405f5f0 con 0x7f61840fb940 2024-03-18T01:13:04.558 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:04.555+0000 7f6172ffd700 1 --2- 172.21.15.90:0/3886463845 >> [v2:172.21.15.29:6800/2687730237,v1:172.21.15.29:6801/2687730237] conn(0x7f616c0368d0 0x7f616c038d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:13:04.558 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:04.555+0000 7f6172ffd700 1 -- 172.21.15.90:0/3886463845 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f6178049be0 con 0x7f61840fb940 2024-03-18T01:13:04.558 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:04.555+0000 7f6181d9b700 1 -- 172.21.15.90:0/3886463845 >> [v2:172.21.15.29:6800/2687730237,v1:172.21.15.29:6801/2687730237] conn(0x7f616c0368d0 msgr2=0x7f616c038d80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6800/2687730237 2024-03-18T01:13:04.558 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:04.555+0000 7f6181d9b700 1 --2- 172.21.15.90:0/3886463845 >> [v2:172.21.15.29:6800/2687730237,v1:172.21.15.29:6801/2687730237] conn(0x7f616c0368d0 0x7f616c038d80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T01:13:04.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:04.560+0000 7f6172ffd700 1 -- 172.21.15.90:0/3886463845 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175520 (secure 0 0 0) 0x7f6178011550 con 0x7f61840fb940 2024-03-18T01:13:04.568 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:13:04 smithi029 ceph-mon[29331]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T01:13:04.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:13:04 smithi029 ceph-mon[29331]: from='mgr.14158 172.21.15.29:0/4059439045' entity='mgr.smithi029.qviwhl' 2024-03-18T01:13:04.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:13:04 smithi029 ceph-mon[29331]: from='mgr.14158 172.21.15.29:0/4059439045' entity='mgr.smithi029.qviwhl' 2024-03-18T01:13:04.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:13:04 smithi029 ceph-mon[29331]: from='mgr.14158 172.21.15.29:0/4059439045' entity='mgr.smithi029.qviwhl' 2024-03-18T01:13:04.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:13:04 smithi029 ceph-mon[29331]: from='mgr.14158 172.21.15.29:0/4059439045' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-03-18T01:13:04.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:04.756+0000 7f6181d9b700 1 -- 172.21.15.90:0/3886463845 >> [v2:172.21.15.29:6800/2687730237,v1:172.21.15.29:6801/2687730237] conn(0x7f616c0368d0 msgr2=0x7f616c038d80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6800/2687730237 2024-03-18T01:13:04.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:04.756+0000 7f6181d9b700 1 --2- 172.21.15.90:0/3886463845 >> [v2:172.21.15.29:6800/2687730237,v1:172.21.15.29:6801/2687730237] conn(0x7f616c0368d0 0x7f616c038d80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T01:13:04.819 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:04.818+0000 7f6188ab3700 1 -- 172.21.15.90:0/3886463845 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f618404bed0 con 0x7f61840fb940 2024-03-18T01:13:04.819 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:04.818+0000 7f6172ffd700 1 -- 172.21.15.90:0/3886463845 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7f61780494d0 con 0x7f61840fb940 2024-03-18T01:13:04.820 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-18T01:13:04.820 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":1,"fsid":"37f51182-e4c4-11ee-95c9-87774f69a715","modified":"2024-03-18T01:10:46.941548Z","created":"2024-03-18T01:10:46.941548Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-18T01:13:04.822 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:04.820+0000 7f6188ab3700 1 -- 172.21.15.90:0/3886463845 >> [v2:172.21.15.29:6800/2687730237,v1:172.21.15.29:6801/2687730237] conn(0x7f616c0368d0 msgr2=0x7f616c038d80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T01:13:04.822 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:04.820+0000 7f6188ab3700 1 --2- 172.21.15.90:0/3886463845 >> [v2:172.21.15.29:6800/2687730237,v1:172.21.15.29:6801/2687730237] conn(0x7f616c0368d0 0x7f616c038d80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:13:04.822 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:04.820+0000 7f6188ab3700 1 -- 172.21.15.90:0/3886463845 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f61840fb940 msgr2=0x7f61840f9000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:13:04.822 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:04.820+0000 7f6188ab3700 1 --2- 172.21.15.90:0/3886463845 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f61840fb940 0x7f61840f9000 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f61780096b0 tx=0x7f6178009e30 comp rx=0 tx=0).stop 2024-03-18T01:13:04.822 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:04.820+0000 7f6188ab3700 1 -- 172.21.15.90:0/3886463845 shutdown_connections 2024-03-18T01:13:04.822 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:04.820+0000 7f6188ab3700 1 --2- 172.21.15.90:0/3886463845 >> [v2:172.21.15.29:6800/2687730237,v1:172.21.15.29:6801/2687730237] conn(0x7f616c0368d0 0x7f616c038d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:13:04.823 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:04.820+0000 7f6188ab3700 1 --2- 172.21.15.90:0/3886463845 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f61840fb940 0x7f61840f9000 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:13:04.823 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:04.820+0000 7f6188ab3700 1 -- 172.21.15.90:0/3886463845 >> 172.21.15.90:0/3886463845 conn(0x7f61840f70c0 msgr2=0x7f6184065d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:13:04.823 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:04.820+0000 7f6188ab3700 1 -- 172.21.15.90:0/3886463845 shutdown_connections 2024-03-18T01:13:04.823 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:04.820+0000 7f6188ab3700 1 -- 172.21.15.90:0/3886463845 wait complete. 2024-03-18T01:13:04.823 INFO:teuthology.orchestra.run.smithi090.stderr:dumped monmap epoch 1 2024-03-18T01:13:05.508 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:13:05 smithi029 ceph-mon[29331]: from='mgr.14158 172.21.15.29:0/4059439045' entity='mgr.smithi029.qviwhl' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-03-18T01:13:05.508 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:13:05 smithi029 ceph-mon[29331]: mgrmap e14: smithi029.qviwhl(active, since 65s) 2024-03-18T01:13:05.508 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:13:05 smithi029 ceph-mon[29331]: from='client.? 172.21.15.90:0/3886463845' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-18T01:13:06.346 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-18T01:13:06.346 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 37f51182-e4c4-11ee-95c9-87774f69a715 -- ceph mon dump -f json 2024-03-18T01:13:06.597 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-18T01:13:06.644 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-18T01:13:07.999 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:07.996+0000 7f5c5a81c700 1 -- 172.21.15.90:0/3970285994 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5c540fb960 msgr2=0x7f5c540fbd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:13:07.999 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:07.996+0000 7f5c5a81c700 1 --2- 172.21.15.90:0/3970285994 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5c540fb960 0x7f5c540fbd70 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f5c440048c0 tx=0x7f5c44009e30 comp rx=0 tx=0).stop 2024-03-18T01:13:07.999 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:07.996+0000 7f5c5a81c700 1 -- 172.21.15.90:0/3970285994 shutdown_connections 2024-03-18T01:13:07.999 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:07.996+0000 7f5c5a81c700 1 --2- 172.21.15.90:0/3970285994 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5c540fb960 0x7f5c540fbd70 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:13:07.999 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:07.996+0000 7f5c5a81c700 1 -- 172.21.15.90:0/3970285994 >> 172.21.15.90:0/3970285994 conn(0x7f5c540f70c0 msgr2=0x7f5c540f94f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:13:07.999 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:07.996+0000 7f5c5a81c700 1 -- 172.21.15.90:0/3970285994 shutdown_connections 2024-03-18T01:13:07.999 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:07.996+0000 7f5c5a81c700 1 -- 172.21.15.90:0/3970285994 wait complete. 2024-03-18T01:13:07.999 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:07.997+0000 7f5c5a81c700 1 Processor -- start 2024-03-18T01:13:07.999 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:07.997+0000 7f5c5a81c700 1 -- start start 2024-03-18T01:13:08.000 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:07.997+0000 7f5c5a81c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5c540fb960 0x7f5c5410d080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:13:08.000 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:07.997+0000 7f5c5a81c700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c5410d5c0 con 0x7f5c540fb960 2024-03-18T01:13:08.000 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:07.997+0000 7f5c53fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5c540fb960 0x7f5c5410d080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:13:08.000 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:07.997+0000 7f5c53fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5c540fb960 0x7f5c5410d080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.90:36366/0 (socket says 172.21.15.90:36366) 2024-03-18T01:13:08.000 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:07.997+0000 7f5c53fff700 1 -- 172.21.15.90:0/2462615154 learned_addr learned my addr 172.21.15.90:0/2462615154 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-18T01:13:08.000 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:07.997+0000 7f5c53fff700 1 -- 172.21.15.90:0/2462615154 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c44005040 con 0x7f5c540fb960 2024-03-18T01:13:08.000 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:07.998+0000 7f5c53fff700 1 --2- 172.21.15.90:0/2462615154 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5c540fb960 0x7f5c5410d080 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f5c44009680 tx=0x7f5c44009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:13:08.000 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:07.998+0000 7f5c50ff9700 1 -- 172.21.15.90:0/2462615154 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5c4400b5c0 con 0x7f5c540fb960 2024-03-18T01:13:08.000 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:07.998+0000 7f5c50ff9700 1 -- 172.21.15.90:0/2462615154 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5c4400be70 con 0x7f5c540fb960 2024-03-18T01:13:08.000 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:07.998+0000 7f5c5a81c700 1 -- 172.21.15.90:0/2462615154 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c5410d7c0 con 0x7f5c540fb960 2024-03-18T01:13:08.000 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:07.998+0000 7f5c50ff9700 1 -- 172.21.15.90:0/2462615154 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5c440135a0 con 0x7f5c540fb960 2024-03-18T01:13:08.000 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:07.998+0000 7f5c5a81c700 1 -- 172.21.15.90:0/2462615154 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c5410e330 con 0x7f5c540fb960 2024-03-18T01:13:08.001 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:07.999+0000 7f5c50ff9700 1 -- 172.21.15.90:0/2462615154 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 14) v1 ==== 43438+0+0 (secure 0 0 0) 0x7f5c44013860 con 0x7f5c540fb960 2024-03-18T01:13:08.001 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:07.999+0000 7f5c5a81c700 1 -- 172.21.15.90:0/2462615154 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c54101e70 con 0x7f5c540fb960 2024-03-18T01:13:08.001 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:07.999+0000 7f5c50ff9700 1 --2- 172.21.15.90:0/2462615154 >> [v2:172.21.15.29:6800/2687730237,v1:172.21.15.29:6801/2687730237] conn(0x7f5c3c0368d0 0x7f5c3c038d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:13:08.001 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:07.999+0000 7f5c50ff9700 1 -- 172.21.15.90:0/2462615154 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f5c44049580 con 0x7f5c540fb960 2024-03-18T01:13:08.001 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:07.999+0000 7f5c537fe700 1 -- 172.21.15.90:0/2462615154 >> [v2:172.21.15.29:6800/2687730237,v1:172.21.15.29:6801/2687730237] conn(0x7f5c3c0368d0 msgr2=0x7f5c3c038d80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6800/2687730237 2024-03-18T01:13:08.001 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:07.999+0000 7f5c537fe700 1 --2- 172.21.15.90:0/2462615154 >> [v2:172.21.15.29:6800/2687730237,v1:172.21.15.29:6801/2687730237] conn(0x7f5c3c0368d0 0x7f5c3c038d80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T01:13:08.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:08.004+0000 7f5c50ff9700 1 -- 172.21.15.90:0/2462615154 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175520 (secure 0 0 0) 0x7f5c4401b390 con 0x7f5c540fb960 2024-03-18T01:13:08.201 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:08.200+0000 7f5c537fe700 1 -- 172.21.15.90:0/2462615154 >> [v2:172.21.15.29:6800/2687730237,v1:172.21.15.29:6801/2687730237] conn(0x7f5c3c0368d0 msgr2=0x7f5c3c038d80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6800/2687730237 2024-03-18T01:13:08.201 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:08.200+0000 7f5c537fe700 1 --2- 172.21.15.90:0/2462615154 >> [v2:172.21.15.29:6800/2687730237,v1:172.21.15.29:6801/2687730237] conn(0x7f5c3c0368d0 0x7f5c3c038d80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T01:13:08.263 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:08.262+0000 7f5c5a81c700 1 -- 172.21.15.90:0/2462615154 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5c5410e570 con 0x7f5c540fb960 2024-03-18T01:13:08.263 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:08.262+0000 7f5c50ff9700 1 -- 172.21.15.90:0/2462615154 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7f5c44004d30 con 0x7f5c540fb960 2024-03-18T01:13:08.263 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-18T01:13:08.263 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":1,"fsid":"37f51182-e4c4-11ee-95c9-87774f69a715","modified":"2024-03-18T01:10:46.941548Z","created":"2024-03-18T01:10:46.941548Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-18T01:13:08.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:08.264+0000 7f5c5a81c700 1 -- 172.21.15.90:0/2462615154 >> [v2:172.21.15.29:6800/2687730237,v1:172.21.15.29:6801/2687730237] conn(0x7f5c3c0368d0 msgr2=0x7f5c3c038d80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T01:13:08.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:08.264+0000 7f5c5a81c700 1 --2- 172.21.15.90:0/2462615154 >> [v2:172.21.15.29:6800/2687730237,v1:172.21.15.29:6801/2687730237] conn(0x7f5c3c0368d0 0x7f5c3c038d80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:13:08.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:08.264+0000 7f5c5a81c700 1 -- 172.21.15.90:0/2462615154 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5c540fb960 msgr2=0x7f5c5410d080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:13:08.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:08.264+0000 7f5c5a81c700 1 --2- 172.21.15.90:0/2462615154 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5c540fb960 0x7f5c5410d080 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f5c44009680 tx=0x7f5c44009e30 comp rx=0 tx=0).stop 2024-03-18T01:13:08.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:08.264+0000 7f5c5a81c700 1 -- 172.21.15.90:0/2462615154 shutdown_connections 2024-03-18T01:13:08.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:08.264+0000 7f5c5a81c700 1 --2- 172.21.15.90:0/2462615154 >> [v2:172.21.15.29:6800/2687730237,v1:172.21.15.29:6801/2687730237] conn(0x7f5c3c0368d0 0x7f5c3c038d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:13:08.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:08.264+0000 7f5c5a81c700 1 --2- 172.21.15.90:0/2462615154 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5c540fb960 0x7f5c5410d080 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:13:08.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:08.264+0000 7f5c5a81c700 1 -- 172.21.15.90:0/2462615154 >> 172.21.15.90:0/2462615154 conn(0x7f5c540f70c0 msgr2=0x7f5c54065d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:13:08.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:08.264+0000 7f5c5a81c700 1 -- 172.21.15.90:0/2462615154 shutdown_connections 2024-03-18T01:13:08.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:08.265+0000 7f5c5a81c700 1 -- 172.21.15.90:0/2462615154 wait complete. 2024-03-18T01:13:08.267 INFO:teuthology.orchestra.run.smithi090.stderr:dumped monmap epoch 1 2024-03-18T01:13:08.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:13:08 smithi029 ceph-mon[29331]: from='client.? 172.21.15.90:0/2462615154' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-18T01:13:09.813 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-18T01:13:09.814 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 37f51182-e4c4-11ee-95c9-87774f69a715 -- ceph mon dump -f json 2024-03-18T01:13:10.074 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-18T01:13:10.125 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-18T01:13:11.251 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:13:11 smithi029 ceph-mon[29331]: Active manager daemon smithi029.qviwhl restarted 2024-03-18T01:13:11.251 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:13:11 smithi029 ceph-mon[29331]: Activating manager daemon smithi029.qviwhl 2024-03-18T01:13:11.251 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:13:11 smithi029 ceph-mon[29331]: osdmap e5: 0 total, 0 up, 0 in 2024-03-18T01:13:11.251 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:13:11 smithi029 ceph-mon[29331]: mgrmap e15: smithi029.qviwhl(active, starting, since 0.00391888s) 2024-03-18T01:13:11.251 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:13:11 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "mon metadata", "id": "smithi029"}]: dispatch 2024-03-18T01:13:11.252 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:13:11 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "mgr metadata", "who": "smithi029.qviwhl", "id": "smithi029.qviwhl"}]: dispatch 2024-03-18T01:13:11.252 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:13:11 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-18T01:13:11.252 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:13:11 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-18T01:13:11.252 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:13:11 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-18T01:13:11.252 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:13:11 smithi029 ceph-mon[29331]: Manager daemon smithi029.qviwhl is now available 2024-03-18T01:13:11.252 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:13:11 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T01:13:11.252 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:13:11 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T01:13:11.252 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:13:11 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:13:11.252 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:13:11 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-18T01:13:11.252 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:13:11 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi029.qviwhl/mirror_snapshot_schedule"}]: dispatch 2024-03-18T01:13:11.252 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:13:11 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi029.qviwhl/trash_purge_schedule"}]: dispatch 2024-03-18T01:13:11.668 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:11.667+0000 7fe52e644700 1 -- 172.21.15.90:0/1143593805 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe5280fb950 msgr2=0x7fe5280fbd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:13:11.668 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:11.667+0000 7fe52e644700 1 --2- 172.21.15.90:0/1143593805 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe5280fb950 0x7fe5280fbd60 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fe5180048c0 tx=0x7fe518009e30 comp rx=0 tx=0).stop 2024-03-18T01:13:11.668 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:11.667+0000 7fe52e644700 1 -- 172.21.15.90:0/1143593805 shutdown_connections 2024-03-18T01:13:11.668 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:11.667+0000 7fe52e644700 1 --2- 172.21.15.90:0/1143593805 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe5280fb950 0x7fe5280fbd60 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:13:11.668 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:11.667+0000 7fe52e644700 1 -- 172.21.15.90:0/1143593805 >> 172.21.15.90:0/1143593805 conn(0x7fe5280f7090 msgr2=0x7fe5280f94e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:13:11.668 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:11.667+0000 7fe52e644700 1 -- 172.21.15.90:0/1143593805 shutdown_connections 2024-03-18T01:13:11.668 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:11.667+0000 7fe52e644700 1 -- 172.21.15.90:0/1143593805 wait complete. 2024-03-18T01:13:11.669 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:11.668+0000 7fe52e644700 1 Processor -- start 2024-03-18T01:13:11.669 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:11.668+0000 7fe52e644700 1 -- start start 2024-03-18T01:13:11.669 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:11.668+0000 7fe52e644700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe5280fb950 0x7fe52810bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:13:11.669 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:11.668+0000 7fe52e644700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe52810c4e0 con 0x7fe5280fb950 2024-03-18T01:13:11.670 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:11.668+0000 7fe527fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe5280fb950 0x7fe52810bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:13:11.670 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:11.669+0000 7fe527fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe5280fb950 0x7fe52810bfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.90:36384/0 (socket says 172.21.15.90:36384) 2024-03-18T01:13:11.670 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:11.669+0000 7fe527fff700 1 -- 172.21.15.90:0/1523086444 learned_addr learned my addr 172.21.15.90:0/1523086444 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-18T01:13:11.670 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:11.669+0000 7fe527fff700 1 -- 172.21.15.90:0/1523086444 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe518005040 con 0x7fe5280fb950 2024-03-18T01:13:11.670 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:11.669+0000 7fe527fff700 1 --2- 172.21.15.90:0/1523086444 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe5280fb950 0x7fe52810bfa0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fe5180096b0 tx=0x7fe518009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:13:11.670 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:11.669+0000 7fe524ff9700 1 -- 172.21.15.90:0/1523086444 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe51800bed0 con 0x7fe5280fb950 2024-03-18T01:13:11.670 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:11.669+0000 7fe52e644700 1 -- 172.21.15.90:0/1523086444 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe52810c6e0 con 0x7fe5280fb950 2024-03-18T01:13:11.670 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:11.669+0000 7fe524ff9700 1 -- 172.21.15.90:0/1523086444 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe51800b5c0 con 0x7fe5280fb950 2024-03-18T01:13:11.670 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:11.669+0000 7fe524ff9700 1 -- 172.21.15.90:0/1523086444 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe51801c5b0 con 0x7fe5280fb950 2024-03-18T01:13:11.670 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:11.669+0000 7fe52e644700 1 -- 172.21.15.90:0/1523086444 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe52810d250 con 0x7fe5280fb950 2024-03-18T01:13:11.671 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:11.670+0000 7fe524ff9700 1 -- 172.21.15.90:0/1523086444 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 15) v1 ==== 43162+0+0 (secure 0 0 0) 0x7fe51801c710 con 0x7fe5280fb950 2024-03-18T01:13:11.671 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:11.670+0000 7fe52e644700 1 -- 172.21.15.90:0/1523086444 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe528101e60 con 0x7fe5280fb950 2024-03-18T01:13:11.671 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:11.670+0000 7fe524ff9700 1 -- 172.21.15.90:0/1523086444 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7fe518049640 con 0x7fe5280fb950 2024-03-18T01:13:11.675 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:11.675+0000 7fe524ff9700 1 -- 172.21.15.90:0/1523086444 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175520 (secure 0 0 0) 0x7fe518023300 con 0x7fe5280fb950 2024-03-18T01:13:11.930 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:11.929+0000 7fe524ff9700 1 -- 172.21.15.90:0/1523086444 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mgrmap(e 16) v1 ==== 43289+0+0 (secure 0 0 0) 0x7fe518004d30 con 0x7fe5280fb950 2024-03-18T01:13:11.930 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:11.929+0000 7fe524ff9700 1 --2- 172.21.15.90:0/1523086444 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7fe510037970 0x7fe510039d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:13:11.930 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:11.930+0000 7fe5277fe700 1 --2- 172.21.15.90:0/1523086444 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7fe510037970 0x7fe510039d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:13:11.930 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:11.930+0000 7fe5277fe700 1 --2- 172.21.15.90:0/1523086444 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7fe510037970 0x7fe510039d50 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fe51c009990 tx=0x7fe51c006e40 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:13:11.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:11.944+0000 7fe52e644700 1 -- 172.21.15.90:0/1523086444 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe52810dc60 con 0x7fe5280fb950 2024-03-18T01:13:11.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:11.945+0000 7fe524ff9700 1 -- 172.21.15.90:0/1523086444 <== mon.0 v2:172.21.15.29:3300/0 8 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7fe518023300 con 0x7fe5280fb950 2024-03-18T01:13:11.945 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-18T01:13:11.945 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":1,"fsid":"37f51182-e4c4-11ee-95c9-87774f69a715","modified":"2024-03-18T01:10:46.941548Z","created":"2024-03-18T01:10:46.941548Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-18T01:13:11.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:11.947+0000 7fe52e644700 1 -- 172.21.15.90:0/1523086444 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7fe510037970 msgr2=0x7fe510039d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:13:11.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:11.947+0000 7fe52e644700 1 --2- 172.21.15.90:0/1523086444 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7fe510037970 0x7fe510039d50 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fe51c009990 tx=0x7fe51c006e40 comp rx=0 tx=0).stop 2024-03-18T01:13:11.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:11.947+0000 7fe52e644700 1 -- 172.21.15.90:0/1523086444 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe5280fb950 msgr2=0x7fe52810bfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:13:11.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:11.947+0000 7fe52e644700 1 --2- 172.21.15.90:0/1523086444 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe5280fb950 0x7fe52810bfa0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fe5180096b0 tx=0x7fe518009e30 comp rx=0 tx=0).stop 2024-03-18T01:13:11.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:11.947+0000 7fe52e644700 1 -- 172.21.15.90:0/1523086444 shutdown_connections 2024-03-18T01:13:11.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:11.947+0000 7fe52e644700 1 --2- 172.21.15.90:0/1523086444 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7fe510037970 0x7fe510039d50 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:13:11.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:11.947+0000 7fe52e644700 1 --2- 172.21.15.90:0/1523086444 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe5280fb950 0x7fe52810bfa0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:13:11.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:11.947+0000 7fe52e644700 1 -- 172.21.15.90:0/1523086444 >> 172.21.15.90:0/1523086444 conn(0x7fe5280f7090 msgr2=0x7fe528102a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:13:11.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:11.947+0000 7fe52e644700 1 -- 172.21.15.90:0/1523086444 shutdown_connections 2024-03-18T01:13:11.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:11.947+0000 7fe52e644700 1 -- 172.21.15.90:0/1523086444 wait complete. 2024-03-18T01:13:11.948 INFO:teuthology.orchestra.run.smithi090.stderr:dumped monmap epoch 1 2024-03-18T01:13:13.069 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:13:12 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:13:13.069 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:13:12 smithi029 ceph-mon[29331]: mgrmap e16: smithi029.qviwhl(active, since 1.00502s) 2024-03-18T01:13:13.069 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:13:12 smithi029 ceph-mon[29331]: from='client.? 172.21.15.90:0/1523086444' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-18T01:13:13.069 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:13:12 smithi029 ceph-mon[29331]: [18/Mar/2024:01:13:12] ENGINE Bus STARTING 2024-03-18T01:13:13.069 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:13:12 smithi029 ceph-mon[29331]: [18/Mar/2024:01:13:12] ENGINE Serving on https://172.21.15.29:7150 2024-03-18T01:13:13.069 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:13:12 smithi029 ceph-mon[29331]: [18/Mar/2024:01:13:12] ENGINE Bus STARTED 2024-03-18T01:13:13.070 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:13:12 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:13:13.070 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:13:12 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:13:13.502 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-18T01:13:13.502 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 37f51182-e4c4-11ee-95c9-87774f69a715 -- ceph mon dump -f json 2024-03-18T01:13:14.486 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-18T01:13:14.536 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-18T01:13:14.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:13:14 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:13:14.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:13:14 smithi029 ceph-mon[29331]: mgrmap e17: smithi029.qviwhl(active, since 2s) 2024-03-18T01:13:17.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:17.185+0000 7f70d866a700 1 -- 172.21.15.90:0/2147047605 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f70d00fca50 msgr2=0x7f70d00fce60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:13:17.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:17.185+0000 7f70d866a700 1 --2- 172.21.15.90:0/2147047605 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f70d00fca50 0x7f70d00fce60 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f70c00048c0 tx=0x7f70c0009e30 comp rx=0 tx=0).stop 2024-03-18T01:13:17.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:17.185+0000 7f70d866a700 1 -- 172.21.15.90:0/2147047605 shutdown_connections 2024-03-18T01:13:17.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:17.185+0000 7f70d866a700 1 --2- 172.21.15.90:0/2147047605 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f70d00fca50 0x7f70d00fce60 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:13:17.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:17.185+0000 7f70d866a700 1 -- 172.21.15.90:0/2147047605 >> 172.21.15.90:0/2147047605 conn(0x7f70d00f8190 msgr2=0x7f70d00fa5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:13:17.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:17.185+0000 7f70d866a700 1 -- 172.21.15.90:0/2147047605 shutdown_connections 2024-03-18T01:13:17.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:17.185+0000 7f70d866a700 1 -- 172.21.15.90:0/2147047605 wait complete. 2024-03-18T01:13:17.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:17.186+0000 7f70d866a700 1 Processor -- start 2024-03-18T01:13:17.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:17.186+0000 7f70d866a700 1 -- start start 2024-03-18T01:13:17.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:17.186+0000 7f70d866a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f70d00fca50 0x7f70d0071d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:13:17.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:17.186+0000 7f70d866a700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70d00722a0 con 0x7f70d00fca50 2024-03-18T01:13:17.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:17.186+0000 7f70d6406700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f70d00fca50 0x7f70d0071d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:13:17.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:17.186+0000 7f70d6406700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f70d00fca50 0x7f70d0071d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.90:35778/0 (socket says 172.21.15.90:35778) 2024-03-18T01:13:17.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:17.186+0000 7f70d6406700 1 -- 172.21.15.90:0/2169000586 learned_addr learned my addr 172.21.15.90:0/2169000586 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-18T01:13:17.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:17.187+0000 7f70d6406700 1 -- 172.21.15.90:0/2169000586 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70c0005040 con 0x7f70d00fca50 2024-03-18T01:13:17.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:17.187+0000 7f70d6406700 1 --2- 172.21.15.90:0/2169000586 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f70d00fca50 0x7f70d0071d60 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f70c00096b0 tx=0x7f70c0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:13:17.189 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:17.187+0000 7f70c6ffd700 1 -- 172.21.15.90:0/2169000586 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f70c000bed0 con 0x7f70d00fca50 2024-03-18T01:13:17.189 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:17.187+0000 7f70d866a700 1 -- 172.21.15.90:0/2169000586 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70d0070370 con 0x7f70d00fca50 2024-03-18T01:13:17.189 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:17.187+0000 7f70c6ffd700 1 -- 172.21.15.90:0/2169000586 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f70c000b5c0 con 0x7f70d00fca50 2024-03-18T01:13:17.189 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:17.187+0000 7f70c6ffd700 1 -- 172.21.15.90:0/2169000586 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f70c001c5b0 con 0x7f70d00fca50 2024-03-18T01:13:17.189 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:17.187+0000 7f70d866a700 1 -- 172.21.15.90:0/2169000586 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70d0070860 con 0x7f70d00fca50 2024-03-18T01:13:17.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:17.188+0000 7f70c6ffd700 1 -- 172.21.15.90:0/2169000586 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 43481+0+0 (secure 0 0 0) 0x7f70c001c710 con 0x7f70d00fca50 2024-03-18T01:13:17.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:17.188+0000 7f70d866a700 1 -- 172.21.15.90:0/2169000586 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70d005f5f0 con 0x7f70d00fca50 2024-03-18T01:13:17.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:17.188+0000 7f70c6ffd700 1 --2- 172.21.15.90:0/2169000586 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f70bc0369c0 0x7f70bc038e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:13:17.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:17.188+0000 7f70c6ffd700 1 -- 172.21.15.90:0/2169000586 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7f70c0049d50 con 0x7f70d00fca50 2024-03-18T01:13:17.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:17.188+0000 7f70d5c05700 1 --2- 172.21.15.90:0/2169000586 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f70bc0369c0 0x7f70bc038e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:13:17.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:17.189+0000 7f70d5c05700 1 --2- 172.21.15.90:0/2169000586 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f70bc0369c0 0x7f70bc038e70 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f70cc009990 tx=0x7f70cc006e40 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:13:17.193 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:17.193+0000 7f70c6ffd700 1 -- 172.21.15.90:0/2169000586 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175520 (secure 0 0 0) 0x7f70c0025400 con 0x7f70d00fca50 2024-03-18T01:13:17.449 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:17.449+0000 7f70d866a700 1 -- 172.21.15.90:0/2169000586 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f70d0106430 con 0x7f70d00fca50 2024-03-18T01:13:17.450 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:17.450+0000 7f70c6ffd700 1 -- 172.21.15.90:0/2169000586 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7f70c0011b60 con 0x7f70d00fca50 2024-03-18T01:13:17.450 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-18T01:13:17.450 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":1,"fsid":"37f51182-e4c4-11ee-95c9-87774f69a715","modified":"2024-03-18T01:10:46.941548Z","created":"2024-03-18T01:10:46.941548Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-18T01:13:17.452 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:17.452+0000 7f70d866a700 1 -- 172.21.15.90:0/2169000586 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f70bc0369c0 msgr2=0x7f70bc038e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:13:17.452 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:17.452+0000 7f70d866a700 1 --2- 172.21.15.90:0/2169000586 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f70bc0369c0 0x7f70bc038e70 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f70cc009990 tx=0x7f70cc006e40 comp rx=0 tx=0).stop 2024-03-18T01:13:17.452 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:17.452+0000 7f70d866a700 1 -- 172.21.15.90:0/2169000586 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f70d00fca50 msgr2=0x7f70d0071d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:13:17.452 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:17.452+0000 7f70d866a700 1 --2- 172.21.15.90:0/2169000586 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f70d00fca50 0x7f70d0071d60 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f70c00096b0 tx=0x7f70c0009e30 comp rx=0 tx=0).stop 2024-03-18T01:13:17.452 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:17.452+0000 7f70d866a700 1 -- 172.21.15.90:0/2169000586 shutdown_connections 2024-03-18T01:13:17.452 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:17.452+0000 7f70d866a700 1 --2- 172.21.15.90:0/2169000586 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f70bc0369c0 0x7f70bc038e70 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:13:17.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:17.452+0000 7f70d866a700 1 --2- 172.21.15.90:0/2169000586 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f70d00fca50 0x7f70d0071d60 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:13:17.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:17.452+0000 7f70d866a700 1 -- 172.21.15.90:0/2169000586 >> 172.21.15.90:0/2169000586 conn(0x7f70d00f8190 msgr2=0x7f70d0103b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:13:17.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:17.452+0000 7f70d866a700 1 -- 172.21.15.90:0/2169000586 shutdown_connections 2024-03-18T01:13:17.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:17.452+0000 7f70d866a700 1 -- 172.21.15.90:0/2169000586 wait complete. 2024-03-18T01:13:17.453 INFO:teuthology.orchestra.run.smithi090.stderr:dumped monmap epoch 1 2024-03-18T01:13:18.315 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:13:17 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:13:18.316 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:13:17 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:13:18.316 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:13:17 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:13:18.316 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:13:17 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:13:18.316 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:13:17 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "config rm", "who": "osd/host:smithi090", "name": "osd_memory_target"}]: dispatch 2024-03-18T01:13:18.316 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:13:17 smithi029 ceph-mon[29331]: from='client.? 172.21.15.90:0/2169000586' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-18T01:13:18.978 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-18T01:13:18.979 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 37f51182-e4c4-11ee-95c9-87774f69a715 -- ceph mon dump -f json 2024-03-18T01:13:19.225 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-18T01:13:19.273 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-18T01:13:20.819 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:13:20 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:13:20.819 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:13:20 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:13:20.819 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:13:20 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:13:20.819 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:13:20 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:13:20.819 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:13:20 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "config rm", "who": "osd/host:smithi029", "name": "osd_memory_target"}]: dispatch 2024-03-18T01:13:20.819 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:13:20 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:13:20.819 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:13:20 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T01:13:20.819 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:13:20 smithi029 ceph-mon[29331]: Updating smithi029:/etc/ceph/ceph.conf 2024-03-18T01:13:20.819 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:13:20 smithi029 ceph-mon[29331]: Updating smithi090:/etc/ceph/ceph.conf 2024-03-18T01:13:20.819 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:13:20 smithi029 ceph-mon[29331]: Updating smithi029:/var/lib/ceph/37f51182-e4c4-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-18T01:13:20.819 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:13:20 smithi029 ceph-mon[29331]: Updating smithi090:/var/lib/ceph/37f51182-e4c4-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-18T01:13:20.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:20.886+0000 7f2516ba4700 1 -- 172.21.15.90:0/3247661824 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f25100fe0d0 msgr2=0x7f25100fe4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:13:20.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:20.886+0000 7f2516ba4700 1 --2- 172.21.15.90:0/3247661824 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f25100fe0d0 0x7f25100fe4e0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f2508009c60 tx=0x7f2508004d10 comp rx=0 tx=0).stop 2024-03-18T01:13:20.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:20.886+0000 7f2516ba4700 1 -- 172.21.15.90:0/3247661824 shutdown_connections 2024-03-18T01:13:20.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:20.886+0000 7f2516ba4700 1 --2- 172.21.15.90:0/3247661824 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f25100fe0d0 0x7f25100fe4e0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:13:20.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:20.886+0000 7f2516ba4700 1 -- 172.21.15.90:0/3247661824 >> 172.21.15.90:0/3247661824 conn(0x7f25100f6870 msgr2=0x7f25100f8ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:13:20.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:20.886+0000 7f2516ba4700 1 -- 172.21.15.90:0/3247661824 shutdown_connections 2024-03-18T01:13:20.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:20.886+0000 7f2516ba4700 1 -- 172.21.15.90:0/3247661824 wait complete. 2024-03-18T01:13:20.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:20.886+0000 7f2516ba4700 1 Processor -- start 2024-03-18T01:13:20.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:20.886+0000 7f2516ba4700 1 -- start start 2024-03-18T01:13:20.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:20.887+0000 7f2516ba4700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f25100fe0d0 0x7f251010d0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:13:20.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:20.887+0000 7f2516ba4700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f251010d620 con 0x7f25100fe0d0 2024-03-18T01:13:20.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:20.887+0000 7f2514940700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f25100fe0d0 0x7f251010d0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:13:20.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:20.887+0000 7f2514940700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f25100fe0d0 0x7f251010d0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.90:35802/0 (socket says 172.21.15.90:35802) 2024-03-18T01:13:20.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:20.887+0000 7f2514940700 1 -- 172.21.15.90:0/1272106501 learned_addr learned my addr 172.21.15.90:0/1272106501 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-18T01:13:20.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:20.887+0000 7f2514940700 1 -- 172.21.15.90:0/1272106501 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2508009940 con 0x7f25100fe0d0 2024-03-18T01:13:20.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:20.887+0000 7f2514940700 1 --2- 172.21.15.90:0/1272106501 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f25100fe0d0 0x7f251010d0e0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f2508004e60 tx=0x7f2508004d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:13:20.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:20.888+0000 7f250d7fa700 1 -- 172.21.15.90:0/1272106501 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2508012410 con 0x7f25100fe0d0 2024-03-18T01:13:20.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:20.888+0000 7f250d7fa700 1 -- 172.21.15.90:0/1272106501 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2508012a40 con 0x7f25100fe0d0 2024-03-18T01:13:20.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:20.888+0000 7f2516ba4700 1 -- 172.21.15.90:0/1272106501 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f251010d820 con 0x7f25100fe0d0 2024-03-18T01:13:20.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:20.888+0000 7f250d7fa700 1 -- 172.21.15.90:0/1272106501 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f250801ba40 con 0x7f25100fe0d0 2024-03-18T01:13:20.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:20.888+0000 7f2516ba4700 1 -- 172.21.15.90:0/1272106501 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f251010e390 con 0x7f25100fe0d0 2024-03-18T01:13:20.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:20.888+0000 7f250d7fa700 1 -- 172.21.15.90:0/1272106501 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 43481+0+0 (secure 0 0 0) 0x7f2508019040 con 0x7f25100fe0d0 2024-03-18T01:13:20.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:20.888+0000 7f2516ba4700 1 -- 172.21.15.90:0/1272106501 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f251010dd10 con 0x7f25100fe0d0 2024-03-18T01:13:20.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:20.889+0000 7f250d7fa700 1 --2- 172.21.15.90:0/1272106501 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f24fc0369c0 0x7f24fc038e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:13:20.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:20.889+0000 7f250d7fa700 1 -- 172.21.15.90:0/1272106501 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7f250804a120 con 0x7f25100fe0d0 2024-03-18T01:13:20.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:20.889+0000 7f250ffff700 1 --2- 172.21.15.90:0/1272106501 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f24fc0369c0 0x7f24fc038e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:13:20.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:20.889+0000 7f250ffff700 1 --2- 172.21.15.90:0/1272106501 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f24fc0369c0 0x7f24fc038e70 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f2500009990 tx=0x7f2500006e40 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:13:20.896 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:20.893+0000 7f250d7fa700 1 -- 172.21.15.90:0/1272106501 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175520 (secure 0 0 0) 0x7f250802b730 con 0x7f25100fe0d0 2024-03-18T01:13:21.181 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:21.180+0000 7f2516ba4700 1 -- 172.21.15.90:0/1272106501 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f251005f640 con 0x7f25100fe0d0 2024-03-18T01:13:21.181 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:21.181+0000 7f250d7fa700 1 -- 172.21.15.90:0/1272106501 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7f2508017020 con 0x7f25100fe0d0 2024-03-18T01:13:21.182 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-18T01:13:21.182 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":1,"fsid":"37f51182-e4c4-11ee-95c9-87774f69a715","modified":"2024-03-18T01:10:46.941548Z","created":"2024-03-18T01:10:46.941548Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-18T01:13:21.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:21.183+0000 7f2516ba4700 1 -- 172.21.15.90:0/1272106501 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f24fc0369c0 msgr2=0x7f24fc038e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:13:21.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:21.183+0000 7f2516ba4700 1 --2- 172.21.15.90:0/1272106501 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f24fc0369c0 0x7f24fc038e70 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f2500009990 tx=0x7f2500006e40 comp rx=0 tx=0).stop 2024-03-18T01:13:21.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:21.183+0000 7f2516ba4700 1 -- 172.21.15.90:0/1272106501 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f25100fe0d0 msgr2=0x7f251010d0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:13:21.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:21.183+0000 7f2516ba4700 1 --2- 172.21.15.90:0/1272106501 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f25100fe0d0 0x7f251010d0e0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f2508004e60 tx=0x7f2508004d10 comp rx=0 tx=0).stop 2024-03-18T01:13:21.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:21.183+0000 7f2516ba4700 1 -- 172.21.15.90:0/1272106501 shutdown_connections 2024-03-18T01:13:21.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:21.183+0000 7f2516ba4700 1 --2- 172.21.15.90:0/1272106501 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f24fc0369c0 0x7f24fc038e70 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:13:21.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:21.183+0000 7f2516ba4700 1 --2- 172.21.15.90:0/1272106501 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f25100fe0d0 0x7f251010d0e0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:13:21.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:21.183+0000 7f2516ba4700 1 -- 172.21.15.90:0/1272106501 >> 172.21.15.90:0/1272106501 conn(0x7f25100f6870 msgr2=0x7f2510103bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:13:21.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:21.183+0000 7f2516ba4700 1 -- 172.21.15.90:0/1272106501 shutdown_connections 2024-03-18T01:13:21.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:21.183+0000 7f2516ba4700 1 -- 172.21.15.90:0/1272106501 wait complete. 2024-03-18T01:13:21.185 INFO:teuthology.orchestra.run.smithi090.stderr:dumped monmap epoch 1 2024-03-18T01:13:22.069 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:13:21 smithi029 ceph-mon[29331]: Updating smithi029:/etc/ceph/ceph.client.admin.keyring 2024-03-18T01:13:22.069 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:13:21 smithi029 ceph-mon[29331]: Updating smithi090:/etc/ceph/ceph.client.admin.keyring 2024-03-18T01:13:22.069 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:13:21 smithi029 ceph-mon[29331]: Updating smithi029:/var/lib/ceph/37f51182-e4c4-11ee-95c9-87774f69a715/config/ceph.client.admin.keyring 2024-03-18T01:13:22.069 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:13:21 smithi029 ceph-mon[29331]: Updating smithi090:/var/lib/ceph/37f51182-e4c4-11ee-95c9-87774f69a715/config/ceph.client.admin.keyring 2024-03-18T01:13:22.069 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:13:21 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:13:22.069 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:13:21 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:13:22.069 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:13:21 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:13:22.070 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:13:21 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:13:22.070 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:13:21 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:13:22.070 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:13:21 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi090", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-03-18T01:13:22.070 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:13:21 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi090", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2024-03-18T01:13:22.070 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:13:21 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:13:22.070 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:13:21 smithi029 ceph-mon[29331]: Deploying daemon crash.smithi090 on smithi090 2024-03-18T01:13:22.070 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:13:21 smithi029 ceph-mon[29331]: from='client.? 172.21.15.90:0/1272106501' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-18T01:13:22.860 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-18T01:13:22.860 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 37f51182-e4c4-11ee-95c9-87774f69a715 -- ceph mon dump -f json 2024-03-18T01:13:23.189 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/37f51182-e4c4-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-18T01:13:25.040 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:25.039+0000 7f1694fec700 1 -- 172.21.15.90:0/316301613 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f16901019b0 msgr2=0x7f1690101d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:13:25.040 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:25.039+0000 7f1694fec700 1 --2- 172.21.15.90:0/316301613 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f16901019b0 0x7f1690101d80 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f16800048c0 tx=0x7f1680009e30 comp rx=0 tx=0).stop 2024-03-18T01:13:25.040 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:25.040+0000 7f1694fec700 1 -- 172.21.15.90:0/316301613 shutdown_connections 2024-03-18T01:13:25.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:25.040+0000 7f1694fec700 1 --2- 172.21.15.90:0/316301613 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f16901019b0 0x7f1690101d80 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:13:25.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:25.040+0000 7f1694fec700 1 -- 172.21.15.90:0/316301613 >> 172.21.15.90:0/316301613 conn(0x7f16900f7510 msgr2=0x7f16900f9920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:13:25.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:25.040+0000 7f1694fec700 1 -- 172.21.15.90:0/316301613 shutdown_connections 2024-03-18T01:13:25.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:25.040+0000 7f1694fec700 1 -- 172.21.15.90:0/316301613 wait complete. 2024-03-18T01:13:25.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:25.040+0000 7f1694fec700 1 Processor -- start 2024-03-18T01:13:25.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:25.040+0000 7f1694fec700 1 -- start start 2024-03-18T01:13:25.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:25.041+0000 7f1694fec700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f16901019b0 0x7f16901130f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:13:25.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:25.041+0000 7f1694fec700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1690113630 con 0x7f16901019b0 2024-03-18T01:13:25.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:25.041+0000 7f168e59c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f16901019b0 0x7f16901130f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:13:25.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:25.041+0000 7f168e59c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f16901019b0 0x7f16901130f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.90:35822/0 (socket says 172.21.15.90:35822) 2024-03-18T01:13:25.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:25.041+0000 7f168e59c700 1 -- 172.21.15.90:0/2007591732 learned_addr learned my addr 172.21.15.90:0/2007591732 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-18T01:13:25.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:25.041+0000 7f168e59c700 1 -- 172.21.15.90:0/2007591732 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1680005040 con 0x7f16901019b0 2024-03-18T01:13:25.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:25.041+0000 7f168e59c700 1 --2- 172.21.15.90:0/2007591732 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f16901019b0 0x7f16901130f0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f168000b7f0 tx=0x7f1680009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:13:25.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:25.042+0000 7f16877fe700 1 -- 172.21.15.90:0/2007591732 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1680009370 con 0x7f16901019b0 2024-03-18T01:13:25.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:25.042+0000 7f1694fec700 1 -- 172.21.15.90:0/2007591732 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1690113830 con 0x7f16901019b0 2024-03-18T01:13:25.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:25.042+0000 7f16877fe700 1 -- 172.21.15.90:0/2007591732 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1680013420 con 0x7f16901019b0 2024-03-18T01:13:25.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:25.042+0000 7f16877fe700 1 -- 172.21.15.90:0/2007591732 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f168001c590 con 0x7f16901019b0 2024-03-18T01:13:25.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:25.042+0000 7f1694fec700 1 -- 172.21.15.90:0/2007591732 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1690113d20 con 0x7f16901019b0 2024-03-18T01:13:25.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:25.043+0000 7f16877fe700 1 -- 172.21.15.90:0/2007591732 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 43481+0+0 (secure 0 0 0) 0x7f1680026440 con 0x7f16901019b0 2024-03-18T01:13:25.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:25.043+0000 7f1694fec700 1 -- 172.21.15.90:0/2007591732 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16900fd0b0 con 0x7f16901019b0 2024-03-18T01:13:25.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:25.043+0000 7f16877fe700 1 --2- 172.21.15.90:0/2007591732 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f167c0369c0 0x7f167c038e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:13:25.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:25.043+0000 7f16877fe700 1 -- 172.21.15.90:0/2007591732 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7f1680049560 con 0x7f16901019b0 2024-03-18T01:13:25.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:25.043+0000 7f1685d9b700 1 --2- 172.21.15.90:0/2007591732 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f167c0369c0 0x7f167c038e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:13:25.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:25.043+0000 7f1685d9b700 1 --2- 172.21.15.90:0/2007591732 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f167c0369c0 0x7f167c038e70 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f1678009990 tx=0x7f1678006e40 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:13:25.048 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:25.048+0000 7f16877fe700 1 -- 172.21.15.90:0/2007591732 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175520 (secure 0 0 0) 0x7f1680011780 con 0x7f16901019b0 2024-03-18T01:13:25.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:25.319+0000 7f1694fec700 1 -- 172.21.15.90:0/2007591732 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f169005f5f0 con 0x7f16901019b0 2024-03-18T01:13:25.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:25.319+0000 7f16877fe700 1 -- 172.21.15.90:0/2007591732 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7f1680011370 con 0x7f16901019b0 2024-03-18T01:13:25.320 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-18T01:13:25.320 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":1,"fsid":"37f51182-e4c4-11ee-95c9-87774f69a715","modified":"2024-03-18T01:10:46.941548Z","created":"2024-03-18T01:10:46.941548Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-18T01:13:25.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:25.322+0000 7f1694fec700 1 -- 172.21.15.90:0/2007591732 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f167c0369c0 msgr2=0x7f167c038e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:13:25.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:25.322+0000 7f1694fec700 1 --2- 172.21.15.90:0/2007591732 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f167c0369c0 0x7f167c038e70 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f1678009990 tx=0x7f1678006e40 comp rx=0 tx=0).stop 2024-03-18T01:13:25.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:25.322+0000 7f1694fec700 1 -- 172.21.15.90:0/2007591732 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f16901019b0 msgr2=0x7f16901130f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:13:25.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:25.322+0000 7f1694fec700 1 --2- 172.21.15.90:0/2007591732 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f16901019b0 0x7f16901130f0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f168000b7f0 tx=0x7f1680009e30 comp rx=0 tx=0).stop 2024-03-18T01:13:25.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:25.322+0000 7f1694fec700 1 -- 172.21.15.90:0/2007591732 shutdown_connections 2024-03-18T01:13:25.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:25.322+0000 7f1694fec700 1 --2- 172.21.15.90:0/2007591732 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f167c0369c0 0x7f167c038e70 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:13:25.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:25.322+0000 7f1694fec700 1 --2- 172.21.15.90:0/2007591732 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f16901019b0 0x7f16901130f0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:13:25.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:25.322+0000 7f1694fec700 1 -- 172.21.15.90:0/2007591732 >> 172.21.15.90:0/2007591732 conn(0x7f16900f7510 msgr2=0x7f1690065ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:13:25.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:25.322+0000 7f1694fec700 1 -- 172.21.15.90:0/2007591732 shutdown_connections 2024-03-18T01:13:25.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:25.322+0000 7f1694fec700 1 -- 172.21.15.90:0/2007591732 wait complete. 2024-03-18T01:13:25.323 INFO:teuthology.orchestra.run.smithi090.stderr:dumped monmap epoch 1 2024-03-18T01:13:26.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:13:26 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:13:26.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:13:26 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:13:26.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:13:26 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:13:26.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:13:26 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:13:26.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:13:26 smithi029 ceph-mon[29331]: Deploying daemon node-exporter.smithi090 on smithi090 2024-03-18T01:13:26.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:13:26 smithi029 ceph-mon[29331]: from='client.? 172.21.15.90:0/2007591732' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-18T01:13:26.570 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:13:26 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:13:26.570 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:13:26 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-18T01:13:26.887 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-18T01:13:26.888 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 37f51182-e4c4-11ee-95c9-87774f69a715 -- ceph mon dump -f json 2024-03-18T01:13:27.137 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/37f51182-e4c4-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-18T01:13:29.602 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:29.540+0000 7f543cfa7700 1 -- 172.21.15.90:0/2460197151 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5438102bd0 msgr2=0x7f5438102fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:13:29.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:29.540+0000 7f543cfa7700 1 --2- 172.21.15.90:0/2460197151 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5438102bd0 0x7f5438102fa0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f542c0048c0 tx=0x7f542c009e30 comp rx=0 tx=0).stop 2024-03-18T01:13:29.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:29.540+0000 7f543cfa7700 1 -- 172.21.15.90:0/2460197151 shutdown_connections 2024-03-18T01:13:29.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:29.540+0000 7f543cfa7700 1 --2- 172.21.15.90:0/2460197151 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5438102bd0 0x7f5438102fa0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:13:29.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:29.540+0000 7f543cfa7700 1 -- 172.21.15.90:0/2460197151 >> 172.21.15.90:0/2460197151 conn(0x7f54380f85e0 msgr2=0x7f54380fa9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:13:29.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:29.540+0000 7f543cfa7700 1 -- 172.21.15.90:0/2460197151 shutdown_connections 2024-03-18T01:13:29.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:29.540+0000 7f543cfa7700 1 -- 172.21.15.90:0/2460197151 wait complete. 2024-03-18T01:13:29.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:29.540+0000 7f543cfa7700 1 Processor -- start 2024-03-18T01:13:29.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:29.541+0000 7f543cfa7700 1 -- start start 2024-03-18T01:13:29.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:29.541+0000 7f543cfa7700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5438102bd0 0x7f5438113230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:13:29.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:29.541+0000 7f543cfa7700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5438113770 con 0x7f5438102bd0 2024-03-18T01:13:29.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:29.541+0000 7f543659c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5438102bd0 0x7f5438113230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:13:29.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:29.541+0000 7f543659c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5438102bd0 0x7f5438113230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.90:36098/0 (socket says 172.21.15.90:36098) 2024-03-18T01:13:29.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:29.541+0000 7f543659c700 1 -- 172.21.15.90:0/1359355591 learned_addr learned my addr 172.21.15.90:0/1359355591 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-18T01:13:29.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:29.541+0000 7f543659c700 1 -- 172.21.15.90:0/1359355591 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f542c005040 con 0x7f5438102bd0 2024-03-18T01:13:29.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:29.541+0000 7f543659c700 1 --2- 172.21.15.90:0/1359355591 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5438102bd0 0x7f5438113230 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f542c00b7f0 tx=0x7f542c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:13:29.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:29.541+0000 7f5426ffd700 1 -- 172.21.15.90:0/1359355591 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f542c009370 con 0x7f5438102bd0 2024-03-18T01:13:29.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:29.542+0000 7f543cfa7700 1 -- 172.21.15.90:0/1359355591 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5438113970 con 0x7f5438102bd0 2024-03-18T01:13:29.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:29.542+0000 7f5426ffd700 1 -- 172.21.15.90:0/1359355591 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f542c013420 con 0x7f5438102bd0 2024-03-18T01:13:29.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:29.542+0000 7f5426ffd700 1 -- 172.21.15.90:0/1359355591 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f542c01c590 con 0x7f5438102bd0 2024-03-18T01:13:29.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:29.542+0000 7f543cfa7700 1 -- 172.21.15.90:0/1359355591 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5438113e60 con 0x7f5438102bd0 2024-03-18T01:13:29.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:29.542+0000 7f543cfa7700 1 -- 172.21.15.90:0/1359355591 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f543805f5f0 con 0x7f5438102bd0 2024-03-18T01:13:29.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:29.542+0000 7f5426ffd700 1 -- 172.21.15.90:0/1359355591 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 43481+0+0 (secure 0 0 0) 0x7f542c013a70 con 0x7f5438102bd0 2024-03-18T01:13:29.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:29.543+0000 7f5426ffd700 1 --2- 172.21.15.90:0/1359355591 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f5420036970 0x7f5420038e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:13:29.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:29.543+0000 7f5426ffd700 1 -- 172.21.15.90:0/1359355591 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7f542c0494b0 con 0x7f5438102bd0 2024-03-18T01:13:29.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:29.543+0000 7f5435d9b700 1 --2- 172.21.15.90:0/1359355591 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f5420036970 0x7f5420038e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:13:29.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:29.544+0000 7f5435d9b700 1 --2- 172.21.15.90:0/1359355591 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f5420036970 0x7f5420038e20 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f5428009990 tx=0x7f5428006e40 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:13:29.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:29.548+0000 7f5426ffd700 1 -- 172.21.15.90:0/1359355591 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175520 (secure 0 0 0) 0x7f542c024d30 con 0x7f5438102bd0 2024-03-18T01:13:29.811 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:29.810+0000 7f543cfa7700 1 -- 172.21.15.90:0/1359355591 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f543810e880 con 0x7f5438102bd0 2024-03-18T01:13:29.811 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:29.811+0000 7f5426ffd700 1 -- 172.21.15.90:0/1359355591 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7f542c021070 con 0x7f5438102bd0 2024-03-18T01:13:29.811 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-18T01:13:29.811 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":1,"fsid":"37f51182-e4c4-11ee-95c9-87774f69a715","modified":"2024-03-18T01:10:46.941548Z","created":"2024-03-18T01:10:46.941548Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-18T01:13:29.813 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:29.813+0000 7f543cfa7700 1 -- 172.21.15.90:0/1359355591 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f5420036970 msgr2=0x7f5420038e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:13:29.813 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:29.813+0000 7f543cfa7700 1 --2- 172.21.15.90:0/1359355591 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f5420036970 0x7f5420038e20 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f5428009990 tx=0x7f5428006e40 comp rx=0 tx=0).stop 2024-03-18T01:13:29.814 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:29.813+0000 7f543cfa7700 1 -- 172.21.15.90:0/1359355591 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5438102bd0 msgr2=0x7f5438113230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:13:29.814 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:29.813+0000 7f543cfa7700 1 --2- 172.21.15.90:0/1359355591 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5438102bd0 0x7f5438113230 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f542c00b7f0 tx=0x7f542c009e30 comp rx=0 tx=0).stop 2024-03-18T01:13:29.814 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:29.813+0000 7f543cfa7700 1 -- 172.21.15.90:0/1359355591 shutdown_connections 2024-03-18T01:13:29.814 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:29.813+0000 7f543cfa7700 1 --2- 172.21.15.90:0/1359355591 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f5420036970 0x7f5420038e20 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:13:29.814 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:29.813+0000 7f543cfa7700 1 --2- 172.21.15.90:0/1359355591 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5438102bd0 0x7f5438113230 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:13:29.814 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:29.813+0000 7f543cfa7700 1 -- 172.21.15.90:0/1359355591 >> 172.21.15.90:0/1359355591 conn(0x7f54380f85e0 msgr2=0x7f54380f9c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:13:29.814 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:29.813+0000 7f543cfa7700 1 -- 172.21.15.90:0/1359355591 shutdown_connections 2024-03-18T01:13:29.814 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:29.813+0000 7f543cfa7700 1 -- 172.21.15.90:0/1359355591 wait complete. 2024-03-18T01:13:29.814 INFO:teuthology.orchestra.run.smithi090.stderr:dumped monmap epoch 1 2024-03-18T01:13:30.319 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:13:29 smithi029 ceph-mon[29331]: from='client.? 172.21.15.90:0/1359355591' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-18T01:13:31.388 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-18T01:13:31.388 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 37f51182-e4c4-11ee-95c9-87774f69a715 -- ceph mon dump -f json 2024-03-18T01:13:32.069 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:13:31 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:13:32.069 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:13:31 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:13:32.069 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:13:31 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:13:32.069 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:13:31 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:13:32.069 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:13:31 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi090.pnqedy", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-18T01:13:32.069 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:13:31 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.smithi090.pnqedy", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-18T01:13:32.069 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:13:31 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-18T01:13:32.069 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:13:31 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:13:32.070 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:13:31 smithi029 ceph-mon[29331]: Deploying daemon mgr.smithi090.pnqedy on smithi090 2024-03-18T01:13:32.070 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:13:31 smithi029 ceph-mon[29331]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T01:13:32.070 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:13:31 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:13:32.180 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/37f51182-e4c4-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-18T01:13:34.036 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:34.035+0000 7f4709614700 1 -- 172.21.15.90:0/2694457935 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f47040fb850 msgr2=0x7f47040fbc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:13:34.036 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:34.035+0000 7f4709614700 1 --2- 172.21.15.90:0/2694457935 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f47040fb850 0x7f47040fbc20 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f46f40048c0 tx=0x7f46f4009e30 comp rx=0 tx=0).stop 2024-03-18T01:13:34.036 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:34.035+0000 7f4709614700 1 -- 172.21.15.90:0/2694457935 shutdown_connections 2024-03-18T01:13:34.036 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:34.035+0000 7f4709614700 1 --2- 172.21.15.90:0/2694457935 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f47040fb850 0x7f47040fbc20 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:13:34.036 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:34.035+0000 7f4709614700 1 -- 172.21.15.90:0/2694457935 >> 172.21.15.90:0/2694457935 conn(0x7f47040f74e0 msgr2=0x7f47040f98f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:13:34.036 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:34.035+0000 7f4709614700 1 -- 172.21.15.90:0/2694457935 shutdown_connections 2024-03-18T01:13:34.036 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:34.035+0000 7f4709614700 1 -- 172.21.15.90:0/2694457935 wait complete. 2024-03-18T01:13:34.038 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:34.036+0000 7f4709614700 1 Processor -- start 2024-03-18T01:13:34.038 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:34.036+0000 7f4709614700 1 -- start start 2024-03-18T01:13:34.038 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:34.036+0000 7f4709614700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f47040fb850 0x7f4704115330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:13:34.038 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:34.036+0000 7f4709614700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4704115870 con 0x7f47040fb850 2024-03-18T01:13:34.038 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:34.037+0000 7f4702ffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f47040fb850 0x7f4704115330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:13:34.038 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:34.037+0000 7f4702ffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f47040fb850 0x7f4704115330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.90:36128/0 (socket says 172.21.15.90:36128) 2024-03-18T01:13:34.038 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:34.037+0000 7f4702ffd700 1 -- 172.21.15.90:0/3828016442 learned_addr learned my addr 172.21.15.90:0/3828016442 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-18T01:13:34.038 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:34.037+0000 7f4702ffd700 1 -- 172.21.15.90:0/3828016442 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46f4005040 con 0x7f47040fb850 2024-03-18T01:13:34.038 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:34.037+0000 7f4702ffd700 1 --2- 172.21.15.90:0/3828016442 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f47040fb850 0x7f4704115330 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f46f400b820 tx=0x7f46f4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:13:34.038 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:34.037+0000 7f46eb7fe700 1 -- 172.21.15.90:0/3828016442 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f46f40093c0 con 0x7f47040fb850 2024-03-18T01:13:34.038 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:34.037+0000 7f46eb7fe700 1 -- 172.21.15.90:0/3828016442 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f46f4013420 con 0x7f47040fb850 2024-03-18T01:13:34.038 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:34.037+0000 7f4709614700 1 -- 172.21.15.90:0/3828016442 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4704115a70 con 0x7f47040fb850 2024-03-18T01:13:34.039 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:34.037+0000 7f46eb7fe700 1 -- 172.21.15.90:0/3828016442 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f46f401c5b0 con 0x7f47040fb850 2024-03-18T01:13:34.039 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:34.037+0000 7f4709614700 1 -- 172.21.15.90:0/3828016442 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4704115f60 con 0x7f47040fb850 2024-03-18T01:13:34.039 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:34.038+0000 7f46eb7fe700 1 -- 172.21.15.90:0/3828016442 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 43481+0+0 (secure 0 0 0) 0x7f46f401c7f0 con 0x7f47040fb850 2024-03-18T01:13:34.039 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:34.038+0000 7f46eb7fe700 1 --2- 172.21.15.90:0/3828016442 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f46ec036970 0x7f46ec038e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:13:34.040 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:34.038+0000 7f46eb7fe700 1 -- 172.21.15.90:0/3828016442 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7f46f404ac00 con 0x7f47040fb850 2024-03-18T01:13:34.040 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:34.039+0000 7f47027fc700 1 --2- 172.21.15.90:0/3828016442 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f46ec036970 0x7f46ec038e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:13:34.040 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:34.039+0000 7f4709614700 1 -- 172.21.15.90:0/3828016442 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47040665a0 con 0x7f47040fb850 2024-03-18T01:13:34.040 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:34.039+0000 7f47027fc700 1 --2- 172.21.15.90:0/3828016442 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f46ec036970 0x7f46ec038e20 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f46f8009990 tx=0x7f46f8006e40 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:13:34.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:34.044+0000 7f46eb7fe700 1 -- 172.21.15.90:0/3828016442 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175520 (secure 0 0 0) 0x7f46f4023300 con 0x7f47040fb850 2024-03-18T01:13:34.319 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:13:33 smithi029 ceph-mon[29331]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T01:13:34.319 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:13:33 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:13:34.319 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:13:33 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:13:34.319 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:13:33 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:13:34.319 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:13:33 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:13:34.319 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:13:33 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-18T01:13:34.319 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:13:33 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:13:34.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:34.339+0000 7f4709614700 1 -- 172.21.15.90:0/3828016442 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f470405f5f0 con 0x7f47040fb850 2024-03-18T01:13:34.340 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:34.340+0000 7f46eb7fe700 1 -- 172.21.15.90:0/3828016442 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7f46f4034090 con 0x7f47040fb850 2024-03-18T01:13:34.340 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-18T01:13:34.340 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":1,"fsid":"37f51182-e4c4-11ee-95c9-87774f69a715","modified":"2024-03-18T01:10:46.941548Z","created":"2024-03-18T01:10:46.941548Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-18T01:13:34.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:34.342+0000 7f4709614700 1 -- 172.21.15.90:0/3828016442 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f46ec036970 msgr2=0x7f46ec038e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:13:34.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:34.342+0000 7f4709614700 1 --2- 172.21.15.90:0/3828016442 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f46ec036970 0x7f46ec038e20 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f46f8009990 tx=0x7f46f8006e40 comp rx=0 tx=0).stop 2024-03-18T01:13:34.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:34.342+0000 7f4709614700 1 -- 172.21.15.90:0/3828016442 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f47040fb850 msgr2=0x7f4704115330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:13:34.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:34.342+0000 7f4709614700 1 --2- 172.21.15.90:0/3828016442 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f47040fb850 0x7f4704115330 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f46f400b820 tx=0x7f46f4009e30 comp rx=0 tx=0).stop 2024-03-18T01:13:34.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:34.342+0000 7f4709614700 1 -- 172.21.15.90:0/3828016442 shutdown_connections 2024-03-18T01:13:34.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:34.342+0000 7f4709614700 1 --2- 172.21.15.90:0/3828016442 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f46ec036970 0x7f46ec038e20 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:13:34.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:34.342+0000 7f4709614700 1 --2- 172.21.15.90:0/3828016442 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f47040fb850 0x7f4704115330 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:13:34.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:34.342+0000 7f4709614700 1 -- 172.21.15.90:0/3828016442 >> 172.21.15.90:0/3828016442 conn(0x7f47040f74e0 msgr2=0x7f47040f7d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:13:34.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:34.342+0000 7f4709614700 1 -- 172.21.15.90:0/3828016442 shutdown_connections 2024-03-18T01:13:34.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:34.342+0000 7f4709614700 1 -- 172.21.15.90:0/3828016442 wait complete. 2024-03-18T01:13:34.343 INFO:teuthology.orchestra.run.smithi090.stderr:dumped monmap epoch 1 2024-03-18T01:13:35.319 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:13:34 smithi029 ceph-mon[29331]: Deploying daemon mon.smithi090 on smithi090 2024-03-18T01:13:35.319 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:13:34 smithi029 ceph-mon[29331]: from='client.? 172.21.15.90:0/3828016442' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-18T01:13:36.016 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-18T01:13:36.016 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 37f51182-e4c4-11ee-95c9-87774f69a715 -- ceph mon dump -f json 2024-03-18T01:13:36.319 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:13:36 smithi029 ceph-mon[29331]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T01:13:36.319 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:13:36 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:13:36.963 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/37f51182-e4c4-11ee-95c9-87774f69a715/mon.smithi090/config 2024-03-18T01:13:38.319 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:13:38 smithi029 ceph-mon[29331]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T01:13:38.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:38.564+0000 7f4e4187b700 1 -- 172.21.15.90:0/3295766850 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4e3c102c30 msgr2=0x7f4e3c103000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:13:38.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:38.564+0000 7f4e4187b700 1 --2- 172.21.15.90:0/3295766850 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4e3c102c30 0x7f4e3c103000 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f4e300048c0 tx=0x7f4e30009e30 comp rx=0 tx=0).stop 2024-03-18T01:13:38.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:38.564+0000 7f4e4187b700 1 -- 172.21.15.90:0/3295766850 shutdown_connections 2024-03-18T01:13:38.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:38.564+0000 7f4e4187b700 1 --2- 172.21.15.90:0/3295766850 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4e3c102c30 0x7f4e3c103000 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:13:38.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:38.564+0000 7f4e4187b700 1 -- 172.21.15.90:0/3295766850 >> 172.21.15.90:0/3295766850 conn(0x7f4e3c072070 msgr2=0x7f4e3c072470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:13:38.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:38.564+0000 7f4e4187b700 1 -- 172.21.15.90:0/3295766850 shutdown_connections 2024-03-18T01:13:38.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:38.564+0000 7f4e4187b700 1 -- 172.21.15.90:0/3295766850 wait complete. 2024-03-18T01:13:38.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:38.565+0000 7f4e4187b700 1 Processor -- start 2024-03-18T01:13:38.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:38.565+0000 7f4e4187b700 1 -- start start 2024-03-18T01:13:38.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:38.565+0000 7f4e4187b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4e3c102c30 0x7f4e3c113240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:13:38.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:38.565+0000 7f4e4187b700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e3c113780 con 0x7f4e3c102c30 2024-03-18T01:13:38.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:38.566+0000 7f4e3affd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4e3c102c30 0x7f4e3c113240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:13:38.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:38.566+0000 7f4e3affd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4e3c102c30 0x7f4e3c113240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.90:47612/0 (socket says 172.21.15.90:47612) 2024-03-18T01:13:38.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:38.566+0000 7f4e3affd700 1 -- 172.21.15.90:0/134752681 learned_addr learned my addr 172.21.15.90:0/134752681 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-18T01:13:38.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:38.566+0000 7f4e3affd700 1 -- 172.21.15.90:0/134752681 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e30005040 con 0x7f4e3c102c30 2024-03-18T01:13:38.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:38.566+0000 7f4e3affd700 1 --2- 172.21.15.90:0/134752681 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4e3c102c30 0x7f4e3c113240 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f4e3000b820 tx=0x7f4e30009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:13:38.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:38.566+0000 7f4e237fe700 1 -- 172.21.15.90:0/134752681 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4e300093f0 con 0x7f4e3c102c30 2024-03-18T01:13:38.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:38.566+0000 7f4e237fe700 1 -- 172.21.15.90:0/134752681 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4e30013420 con 0x7f4e3c102c30 2024-03-18T01:13:38.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:38.566+0000 7f4e237fe700 1 -- 172.21.15.90:0/134752681 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4e3001c570 con 0x7f4e3c102c30 2024-03-18T01:13:38.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:38.567+0000 7f4e4187b700 1 -- 172.21.15.90:0/134752681 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e3c113980 con 0x7f4e3c102c30 2024-03-18T01:13:38.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:38.567+0000 7f4e4187b700 1 -- 172.21.15.90:0/134752681 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e3c113df0 con 0x7f4e3c102c30 2024-03-18T01:13:38.569 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:38.568+0000 7f4e237fe700 1 -- 172.21.15.90:0/134752681 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 43481+0+0 (secure 0 0 0) 0x7f4e30009550 con 0x7f4e3c102c30 2024-03-18T01:13:38.569 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:38.568+0000 7f4e4187b700 1 -- 172.21.15.90:0/134752681 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e3c0fe490 con 0x7f4e3c102c30 2024-03-18T01:13:38.569 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:38.568+0000 7f4e237fe700 1 --2- 172.21.15.90:0/134752681 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f4e24036970 0x7f4e24038e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:13:38.569 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:38.568+0000 7f4e237fe700 1 -- 172.21.15.90:0/134752681 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7f4e30049b20 con 0x7f4e3c102c30 2024-03-18T01:13:38.570 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:38.568+0000 7f4e3a7fc700 1 --2- 172.21.15.90:0/134752681 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f4e24036970 0x7f4e24038e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:13:38.570 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:38.568+0000 7f4e3a7fc700 1 --2- 172.21.15.90:0/134752681 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f4e24036970 0x7f4e24038e20 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f4e2c009990 tx=0x7f4e2c006e40 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:13:38.573 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:38.573+0000 7f4e237fe700 1 -- 172.21.15.90:0/134752681 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175520 (secure 0 0 0) 0x7f4e30013a70 con 0x7f4e3c102c30 2024-03-18T01:13:38.845 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:38.844+0000 7f4e4187b700 1 -- 172.21.15.90:0/134752681 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4e3c05f5f0 con 0x7f4e3c102c30 2024-03-18T01:13:38.845 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:38.845+0000 7f4e237fe700 1 -- 172.21.15.90:0/134752681 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7f4e30011370 con 0x7f4e3c102c30 2024-03-18T01:13:38.846 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-18T01:13:38.846 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":1,"fsid":"37f51182-e4c4-11ee-95c9-87774f69a715","modified":"2024-03-18T01:10:46.941548Z","created":"2024-03-18T01:10:46.941548Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-18T01:13:38.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:38.848+0000 7f4e4187b700 1 -- 172.21.15.90:0/134752681 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f4e24036970 msgr2=0x7f4e24038e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:13:38.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:38.848+0000 7f4e4187b700 1 --2- 172.21.15.90:0/134752681 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f4e24036970 0x7f4e24038e20 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f4e2c009990 tx=0x7f4e2c006e40 comp rx=0 tx=0).stop 2024-03-18T01:13:38.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:38.848+0000 7f4e4187b700 1 -- 172.21.15.90:0/134752681 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4e3c102c30 msgr2=0x7f4e3c113240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:13:38.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:38.848+0000 7f4e4187b700 1 --2- 172.21.15.90:0/134752681 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4e3c102c30 0x7f4e3c113240 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f4e3000b820 tx=0x7f4e30009e30 comp rx=0 tx=0).stop 2024-03-18T01:13:38.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:38.848+0000 7f4e4187b700 1 -- 172.21.15.90:0/134752681 shutdown_connections 2024-03-18T01:13:38.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:38.848+0000 7f4e4187b700 1 --2- 172.21.15.90:0/134752681 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f4e24036970 0x7f4e24038e20 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:13:38.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:38.848+0000 7f4e4187b700 1 --2- 172.21.15.90:0/134752681 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4e3c102c30 0x7f4e3c113240 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:13:38.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:38.848+0000 7f4e4187b700 1 -- 172.21.15.90:0/134752681 >> 172.21.15.90:0/134752681 conn(0x7f4e3c072070 msgr2=0x7f4e3c103940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:13:38.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:38.848+0000 7f4e4187b700 1 -- 172.21.15.90:0/134752681 shutdown_connections 2024-03-18T01:13:38.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:38.848+0000 7f4e4187b700 1 -- 172.21.15.90:0/134752681 wait complete. 2024-03-18T01:13:38.850 INFO:teuthology.orchestra.run.smithi090.stderr:dumped monmap epoch 1 2024-03-18T01:13:40.693 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-18T01:13:40.693 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 37f51182-e4c4-11ee-95c9-87774f69a715 -- ceph mon dump -f json 2024-03-18T01:13:40.941 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/37f51182-e4c4-11ee-95c9-87774f69a715/mon.smithi090/config 2024-03-18T01:13:44.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:13:44 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:13:44.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:13:44 smithi029 ceph-mon[29331]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T01:13:44.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:13:44 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "mon metadata", "id": "smithi029"}]: dispatch 2024-03-18T01:13:44.570 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:13:44 smithi029 ceph-mon[29331]: mon.smithi029 calling monitor election 2024-03-18T01:13:44.570 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:13:44 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "mon metadata", "id": "smithi090"}]: dispatch 2024-03-18T01:13:44.570 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:13:44 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "mon metadata", "id": "smithi090"}]: dispatch 2024-03-18T01:13:44.570 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:13:44 smithi029 ceph-mon[29331]: from='mgr.? 172.21.15.90:0/1810571580' entity='mgr.smithi090.pnqedy' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi090.pnqedy/crt"}]: dispatch 2024-03-18T01:13:44.570 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:13:44 smithi029 ceph-mon[29331]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T01:13:44.570 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:13:44 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-18T01:13:44.570 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:13:44 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "mon metadata", "id": "smithi090"}]: dispatch 2024-03-18T01:13:44.570 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:13:44 smithi029 ceph-mon[29331]: mon.smithi090 calling monitor election 2024-03-18T01:13:44.570 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:13:44 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "mon metadata", "id": "smithi090"}]: dispatch 2024-03-18T01:13:44.570 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:13:44 smithi029 ceph-mon[29331]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T01:13:44.570 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:13:44 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "mon metadata", "id": "smithi090"}]: dispatch 2024-03-18T01:13:44.570 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:13:44 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "mon metadata", "id": "smithi090"}]: dispatch 2024-03-18T01:13:44.571 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:13:44 smithi029 ceph-mon[29331]: mon.smithi029 is new leader, mons smithi029,smithi090 in quorum (ranks 0,1) 2024-03-18T01:13:44.571 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:13:44 smithi029 ceph-mon[29331]: monmap e2: 2 mons at {smithi029=[v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0],smithi090=[v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-18T01:13:44.571 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:13:44 smithi029 ceph-mon[29331]: fsmap 2024-03-18T01:13:44.571 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:13:44 smithi029 ceph-mon[29331]: osdmap e5: 0 total, 0 up, 0 in 2024-03-18T01:13:44.571 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:13:44 smithi029 ceph-mon[29331]: mgrmap e17: smithi029.qviwhl(active, since 33s) 2024-03-18T01:13:44.571 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:13:44 smithi029 ceph-mon[29331]: Standby manager daemon smithi090.pnqedy started 2024-03-18T01:13:44.571 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:13:44 smithi029 ceph-mon[29331]: from='mgr.? 172.21.15.90:0/1810571580' entity='mgr.smithi090.pnqedy' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-18T01:13:44.571 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:13:44 smithi029 ceph-mon[29331]: overall HEALTH_OK 2024-03-18T01:13:44.571 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:13:44 smithi029 ceph-mon[29331]: from='mgr.? 172.21.15.90:0/1810571580' entity='mgr.smithi090.pnqedy' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi090.pnqedy/key"}]: dispatch 2024-03-18T01:13:44.571 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:13:44 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:13:44.571 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:13:44 smithi029 ceph-mon[29331]: from='mgr.? 172.21.15.90:0/1810571580' entity='mgr.smithi090.pnqedy' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-18T01:13:44.571 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:13:44 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:13:44.571 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:13:44 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:13:44.571 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:13:44 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:13:44.571 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:13:44 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:13:45.436 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:13:45 smithi090 ceph-mon[36989]: mgrmap e18: smithi029.qviwhl(active, since 33s), standbys: smithi090.pnqedy 2024-03-18T01:13:45.436 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:13:45 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "mgr metadata", "who": "smithi090.pnqedy", "id": "smithi090.pnqedy"}]: dispatch 2024-03-18T01:13:45.436 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:13:45 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T01:13:45.436 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:13:45 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "mon metadata", "id": "smithi090"}]: dispatch 2024-03-18T01:13:45.452 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:45.451+0000 7f7ae5139700 1 -- 172.21.15.90:0/641190114 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7ac8003580 msgr2=0x7f7ac8005a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:13:45.452 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:45.451+0000 7f7ae5139700 1 --2- 172.21.15.90:0/641190114 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7ac8003580 0x7f7ac8005a00 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f7ae0074200 tx=0x7f7ad000b040 comp rx=0 tx=0).stop 2024-03-18T01:13:45.452 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:45.451+0000 7f7ae5139700 1 -- 172.21.15.90:0/641190114 shutdown_connections 2024-03-18T01:13:45.452 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:45.451+0000 7f7ae5139700 1 --2- 172.21.15.90:0/641190114 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7ac8003580 0x7f7ac8005a00 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:13:45.452 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:45.451+0000 7f7ae5139700 1 -- 172.21.15.90:0/641190114 >> 172.21.15.90:0/641190114 conn(0x7f7ae0072030 msgr2=0x7f7ae0072430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:13:45.452 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:45.451+0000 7f7ae5139700 1 -- 172.21.15.90:0/641190114 shutdown_connections 2024-03-18T01:13:45.452 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:45.451+0000 7f7ae5139700 1 -- 172.21.15.90:0/641190114 wait complete. 2024-03-18T01:13:45.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:45.452+0000 7f7ae5139700 1 Processor -- start 2024-03-18T01:13:45.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:45.452+0000 7f7ae5139700 1 -- start start 2024-03-18T01:13:45.455 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:45.452+0000 7f7ae5139700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7ac8003580 0x7f7ae0113280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:13:45.455 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:45.452+0000 7f7ae5139700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7ae01137c0 0x7f7ae010d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:13:45.455 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:45.452+0000 7f7ae5139700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ae0113dd0 con 0x7f7ae01137c0 2024-03-18T01:13:45.455 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:45.452+0000 7f7ae5139700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ae010d720 con 0x7f7ac8003580 2024-03-18T01:13:45.455 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:45.453+0000 7f7aded9d700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7ac8003580 0x7f7ae0113280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:13:45.455 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:45.453+0000 7f7aded9d700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7ac8003580 0x7f7ae0113280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:44748/0 (socket says 172.21.15.90:44748) 2024-03-18T01:13:45.455 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:45.453+0000 7f7aded9d700 1 -- 172.21.15.90:0/2884414688 learned_addr learned my addr 172.21.15.90:0/2884414688 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-18T01:13:45.455 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:45.453+0000 7f7ade59c700 1 --2- 172.21.15.90:0/2884414688 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7ae01137c0 0x7f7ae010d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:13:45.455 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:45.453+0000 7f7aded9d700 1 -- 172.21.15.90:0/2884414688 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7ac8003580 msgr2=0x7f7ae0113280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-03-18T01:13:45.455 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:45.453+0000 7f7aded9d700 1 -- 172.21.15.90:0/2884414688 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7ac8003580 msgr2=0x7f7ae0113280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-18T01:13:45.455 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:45.453+0000 7f7aded9d700 1 --2- 172.21.15.90:0/2884414688 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7ac8003580 0x7f7ae0113280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-03-18T01:13:45.455 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:45.453+0000 7f7aded9d700 1 --2- 172.21.15.90:0/2884414688 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7ac8003580 0x7f7ae0113280 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T01:13:45.455 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:45.453+0000 7f7ade59c700 1 -- 172.21.15.90:0/2884414688 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7ac8003580 msgr2=0x7f7ae0113280 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T01:13:45.455 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:45.453+0000 7f7ade59c700 1 --2- 172.21.15.90:0/2884414688 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7ac8003580 0x7f7ae0113280 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:13:45.455 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:45.453+0000 7f7ade59c700 1 -- 172.21.15.90:0/2884414688 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ad0005040 con 0x7f7ae01137c0 2024-03-18T01:13:45.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:45.453+0000 7f7ade59c700 1 --2- 172.21.15.90:0/2884414688 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7ae01137c0 0x7f7ae010d1e0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f7ad400b790 tx=0x7f7ad40091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:13:45.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:45.453+0000 7f7ac77fe700 1 -- 172.21.15.90:0/2884414688 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7ad4011810 con 0x7f7ae01137c0 2024-03-18T01:13:45.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:45.453+0000 7f7ac77fe700 1 -- 172.21.15.90:0/2884414688 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7ad4011e40 con 0x7f7ae01137c0 2024-03-18T01:13:45.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:45.453+0000 7f7ae5139700 1 -- 172.21.15.90:0/2884414688 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ae010da00 con 0x7f7ae01137c0 2024-03-18T01:13:45.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:45.453+0000 7f7ac77fe700 1 -- 172.21.15.90:0/2884414688 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7ad400f2e0 con 0x7f7ae01137c0 2024-03-18T01:13:45.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:45.453+0000 7f7ae5139700 1 -- 172.21.15.90:0/2884414688 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ae010df40 con 0x7f7ae01137c0 2024-03-18T01:13:45.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:45.454+0000 7f7ae5139700 1 -- 172.21.15.90:0/2884414688 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ae0107300 con 0x7f7ae01137c0 2024-03-18T01:13:45.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:45.455+0000 7f7ac77fe700 1 -- 172.21.15.90:0/2884414688 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 18) v1 ==== 86565+0+0 (secure 0 0 0) 0x7f7ad4010380 con 0x7f7ae01137c0 2024-03-18T01:13:45.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:45.455+0000 7f7ac77fe700 1 --2- 172.21.15.90:0/2884414688 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f7acc069360 0x7f7acc06b810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:13:45.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:45.455+0000 7f7ac77fe700 1 -- 172.21.15.90:0/2884414688 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7f7ad4086370 con 0x7f7ae01137c0 2024-03-18T01:13:45.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:45.455+0000 7f7aded9d700 1 --2- 172.21.15.90:0/2884414688 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f7acc069360 0x7f7acc06b810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:13:45.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:45.456+0000 7f7aded9d700 1 --2- 172.21.15.90:0/2884414688 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f7acc069360 0x7f7acc06b810 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f7ad0016920 tx=0x7f7ad000b040 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:13:45.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:45.460+0000 7f7ac77fe700 1 -- 172.21.15.90:0/2884414688 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175520 (secure 0 0 0) 0x7f7ad40570f0 con 0x7f7ae01137c0 2024-03-18T01:13:45.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:13:45 smithi029 ceph-mon[29331]: mgrmap e18: smithi029.qviwhl(active, since 33s), standbys: smithi090.pnqedy 2024-03-18T01:13:45.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:13:45 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "mgr metadata", "who": "smithi090.pnqedy", "id": "smithi090.pnqedy"}]: dispatch 2024-03-18T01:13:45.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:13:45 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T01:13:45.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:13:45 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "mon metadata", "id": "smithi090"}]: dispatch 2024-03-18T01:13:45.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:45.722+0000 7f7ae5139700 1 -- 172.21.15.90:0/2884414688 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7ae010e930 con 0x7f7ae01137c0 2024-03-18T01:13:45.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:45.723+0000 7f7ac77fe700 1 -- 172.21.15.90:0/2884414688 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) v1 ==== 95+0+1013 (secure 0 0 0) 0x7f7ad4015410 con 0x7f7ae01137c0 2024-03-18T01:13:45.723 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-18T01:13:45.723 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":2,"fsid":"37f51182-e4c4-11ee-95c9-87774f69a715","modified":"2024-03-18T01:13:39.018696Z","created":"2024-03-18T01:10:46.941548Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"smithi090","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:3300","nonce":0},{"type":"v1","addr":"172.21.15.90:6789","nonce":0}]},"addr":"172.21.15.90:6789/0","public_addr":"172.21.15.90:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-03-18T01:13:45.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:45.726+0000 7f7ae5139700 1 -- 172.21.15.90:0/2884414688 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f7acc069360 msgr2=0x7f7acc06b810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:13:45.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:45.726+0000 7f7ae5139700 1 --2- 172.21.15.90:0/2884414688 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f7acc069360 0x7f7acc06b810 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f7ad0016920 tx=0x7f7ad000b040 comp rx=0 tx=0).stop 2024-03-18T01:13:45.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:45.726+0000 7f7ae5139700 1 -- 172.21.15.90:0/2884414688 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7ae01137c0 msgr2=0x7f7ae010d1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:13:45.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:45.726+0000 7f7ae5139700 1 --2- 172.21.15.90:0/2884414688 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7ae01137c0 0x7f7ae010d1e0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f7ad400b790 tx=0x7f7ad40091a0 comp rx=0 tx=0).stop 2024-03-18T01:13:45.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:45.726+0000 7f7ae5139700 1 -- 172.21.15.90:0/2884414688 shutdown_connections 2024-03-18T01:13:45.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:45.726+0000 7f7ae5139700 1 --2- 172.21.15.90:0/2884414688 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f7acc069360 0x7f7acc06b810 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:13:45.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:45.726+0000 7f7ae5139700 1 --2- 172.21.15.90:0/2884414688 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7ac8003580 0x7f7ae0113280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:13:45.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:45.726+0000 7f7ae5139700 1 --2- 172.21.15.90:0/2884414688 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7ae01137c0 0x7f7ae010d1e0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:13:45.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:45.726+0000 7f7ae5139700 1 -- 172.21.15.90:0/2884414688 >> 172.21.15.90:0/2884414688 conn(0x7f7ae0072030 msgr2=0x7f7ae00fa650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:13:45.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:45.726+0000 7f7ae5139700 1 -- 172.21.15.90:0/2884414688 shutdown_connections 2024-03-18T01:13:45.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:13:45.726+0000 7f7ae5139700 1 -- 172.21.15.90:0/2884414688 wait complete. 2024-03-18T01:13:45.727 INFO:teuthology.orchestra.run.smithi090.stderr:dumped monmap epoch 2 2024-03-18T01:13:46.310 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:13:46 smithi090 ceph-mon[36989]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T01:13:46.311 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:13:46 smithi090 ceph-mon[36989]: from='client.? 172.21.15.90:0/2884414688' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-18T01:13:46.311 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:13:46 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:13:46.334 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-03-18T01:13:46.335 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 37f51182-e4c4-11ee-95c9-87774f69a715 -- ceph config generate-minimal-conf 2024-03-18T01:13:46.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:13:46 smithi029 ceph-mon[29331]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T01:13:46.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:13:46 smithi029 ceph-mon[29331]: from='client.? 172.21.15.90:0/2884414688' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-18T01:13:46.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:13:46 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:13:46.614 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/37f51182-e4c4-11ee-95c9-87774f69a715/mon.smithi029/config 2024-03-18T01:13:47.758 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:13:47 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:13:47.758 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:13:47 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:13:47.759 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:13:47 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:13:47.759 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:13:47 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T01:13:47.759 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:13:47 smithi029 ceph-mon[29331]: Updating smithi029:/etc/ceph/ceph.conf 2024-03-18T01:13:47.759 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:13:47 smithi029 ceph-mon[29331]: Updating smithi090:/etc/ceph/ceph.conf 2024-03-18T01:13:47.759 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:13:47 smithi029 ceph-mon[29331]: Updating smithi029:/var/lib/ceph/37f51182-e4c4-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-18T01:13:47.759 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:13:47 smithi029 ceph-mon[29331]: Updating smithi090:/var/lib/ceph/37f51182-e4c4-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-18T01:13:47.760 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:13:47 smithi029 ceph-mon[29331]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T01:13:47.760 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:13:47 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:13:47.760 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:13:47 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:13:47.760 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:13:47 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:13:47.760 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:13:47 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:13:47.760 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:13:47 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:13:47.760 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:13:47 smithi029 ceph-mon[29331]: Reconfiguring mon.smithi029 (unknown last config time)... 2024-03-18T01:13:47.761 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:13:47 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-18T01:13:47.761 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:13:47 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-18T01:13:47.761 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:13:47 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:13:47.761 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:13:47 smithi029 ceph-mon[29331]: Reconfiguring daemon mon.smithi029 on smithi029 2024-03-18T01:13:47.776 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:13:47 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:13:47.776 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:13:47 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:13:47.776 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:13:47 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:13:47.776 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:13:47 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T01:13:47.777 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:13:47 smithi090 ceph-mon[36989]: Updating smithi029:/etc/ceph/ceph.conf 2024-03-18T01:13:47.777 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:13:47 smithi090 ceph-mon[36989]: Updating smithi090:/etc/ceph/ceph.conf 2024-03-18T01:13:47.777 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:13:47 smithi090 ceph-mon[36989]: Updating smithi029:/var/lib/ceph/37f51182-e4c4-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-18T01:13:47.777 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:13:47 smithi090 ceph-mon[36989]: Updating smithi090:/var/lib/ceph/37f51182-e4c4-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-18T01:13:47.777 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:13:47 smithi090 ceph-mon[36989]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T01:13:47.777 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:13:47 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:13:47.777 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:13:47 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:13:47.778 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:13:47 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:13:47.778 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:13:47 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:13:47.778 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:13:47 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:13:47.778 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:13:47 smithi090 ceph-mon[36989]: Reconfiguring mon.smithi029 (unknown last config time)... 2024-03-18T01:13:47.778 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:13:47 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-18T01:13:47.778 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:13:47 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-18T01:13:47.778 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:13:47 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:13:47.779 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:13:47 smithi090 ceph-mon[36989]: Reconfiguring daemon mon.smithi029 on smithi029 2024-03-18T01:13:49.819 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:13:49 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:13:49.819 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:13:49 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:13:49.819 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:13:49 smithi029 ceph-mon[29331]: Reconfiguring mgr.smithi029.qviwhl (unknown last config time)... 2024-03-18T01:13:49.819 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:13:49 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi029.qviwhl", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-18T01:13:49.819 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:13:49 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-18T01:13:49.819 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:13:49 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:13:49.819 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:13:49 smithi029 ceph-mon[29331]: Reconfiguring daemon mgr.smithi029.qviwhl on smithi029 2024-03-18T01:13:49.819 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:13:49 smithi029 ceph-mon[29331]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T01:13:50.026 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:13:49 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:13:50.026 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:13:49 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:13:50.026 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:13:49 smithi090 ceph-mon[36989]: Reconfiguring mgr.smithi029.qviwhl (unknown last config time)... 2024-03-18T01:13:50.026 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:13:49 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi029.qviwhl", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-18T01:13:50.026 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:13:49 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-18T01:13:50.026 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:13:49 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:13:50.026 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:13:49 smithi090 ceph-mon[36989]: Reconfiguring daemon mgr.smithi029.qviwhl on smithi029 2024-03-18T01:13:50.026 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:13:49 smithi090 ceph-mon[36989]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T01:13:50.153 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:13:50.153+0000 7f73df5bd700 1 -- 172.21.15.29:0/3053130018 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f73d81026c0 msgr2=0x7f73d80fbd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:13:50.154 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:13:50.153+0000 7f73df5bd700 1 --2- 172.21.15.29:0/3053130018 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f73d81026c0 0x7f73d80fbd30 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f73c8004940 tx=0x7f73c8009e30 comp rx=0 tx=0).stop 2024-03-18T01:13:50.154 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:13:50.153+0000 7f73df5bd700 1 -- 172.21.15.29:0/3053130018 shutdown_connections 2024-03-18T01:13:50.154 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:13:50.153+0000 7f73df5bd700 1 --2- 172.21.15.29:0/3053130018 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f73d81026c0 0x7f73d80fbd30 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:13:50.154 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:13:50.153+0000 7f73df5bd700 1 --2- 172.21.15.29:0/3053130018 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f73d8101d20 0x7f73d81020f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:13:50.154 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:13:50.153+0000 7f73df5bd700 1 -- 172.21.15.29:0/3053130018 >> 172.21.15.29:0/3053130018 conn(0x7f73d80f7870 msgr2=0x7f73d80f9c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:13:50.154 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:13:50.153+0000 7f73df5bd700 1 -- 172.21.15.29:0/3053130018 shutdown_connections 2024-03-18T01:13:50.154 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:13:50.153+0000 7f73df5bd700 1 -- 172.21.15.29:0/3053130018 wait complete. 2024-03-18T01:13:50.156 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:13:50.154+0000 7f73df5bd700 1 Processor -- start 2024-03-18T01:13:50.156 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:13:50.154+0000 7f73df5bd700 1 -- start start 2024-03-18T01:13:50.156 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:13:50.154+0000 7f73df5bd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f73d8101d20 0x7f73d8115bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:13:50.156 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:13:50.154+0000 7f73df5bd700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f73d81026c0 0x7f73d81160f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:13:50.156 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:13:50.154+0000 7f73df5bd700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73d81167d0 con 0x7f73d8101d20 2024-03-18T01:13:50.157 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:13:50.154+0000 7f73df5bd700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73d810fd20 con 0x7f73d81026c0 2024-03-18T01:13:50.157 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:13:50.154+0000 7f73dd359700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f73d8101d20 0x7f73d8115bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:13:50.157 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:13:50.154+0000 7f73dd359700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f73d8101d20 0x7f73d8115bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33816/0 (socket says 172.21.15.29:33816) 2024-03-18T01:13:50.157 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:13:50.154+0000 7f73dd359700 1 -- 172.21.15.29:0/3491273030 learned_addr learned my addr 172.21.15.29:0/3491273030 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-18T01:13:50.157 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:13:50.154+0000 7f73dcb58700 1 --2- 172.21.15.29:0/3491273030 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f73d81026c0 0x7f73d81160f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:13:50.157 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:13:50.154+0000 7f73dd359700 1 -- 172.21.15.29:0/3491273030 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f73d81026c0 msgr2=0x7f73d81160f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:13:50.157 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:13:50.154+0000 7f73dd359700 1 --2- 172.21.15.29:0/3491273030 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f73d81026c0 0x7f73d81160f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:13:50.157 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:13:50.154+0000 7f73dd359700 1 -- 172.21.15.29:0/3491273030 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73c8005040 con 0x7f73d8101d20 2024-03-18T01:13:50.157 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:13:50.155+0000 7f73dd359700 1 --2- 172.21.15.29:0/3491273030 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f73d8101d20 0x7f73d8115bb0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f73d400b790 tx=0x7f73d4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:13:50.157 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:13:50.155+0000 7f73cdffb700 1 -- 172.21.15.29:0/3491273030 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f73d400f770 con 0x7f73d8101d20 2024-03-18T01:13:50.157 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:13:50.155+0000 7f73cdffb700 1 -- 172.21.15.29:0/3491273030 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f73d400fda0 con 0x7f73d8101d20 2024-03-18T01:13:50.157 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:13:50.155+0000 7f73df5bd700 1 -- 172.21.15.29:0/3491273030 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73d810ffa0 con 0x7f73d8101d20 2024-03-18T01:13:50.157 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:13:50.155+0000 7f73cdffb700 1 -- 172.21.15.29:0/3491273030 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f73d400d310 con 0x7f73d8101d20 2024-03-18T01:13:50.157 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:13:50.155+0000 7f73df5bd700 1 -- 172.21.15.29:0/3491273030 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73d8110560 con 0x7f73d8101d20 2024-03-18T01:13:50.157 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:13:50.156+0000 7f73df5bd700 1 -- 172.21.15.29:0/3491273030 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73d80fd470 con 0x7f73d8101d20 2024-03-18T01:13:50.158 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:13:50.156+0000 7f73cdffb700 1 -- 172.21.15.29:0/3491273030 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 18) v1 ==== 86565+0+0 (secure 0 0 0) 0x7f73d4010420 con 0x7f73d8101d20 2024-03-18T01:13:50.158 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:13:50.156+0000 7f73cdffb700 1 --2- 172.21.15.29:0/3491273030 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f73c4069210 0x7f73c406b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:13:50.158 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:13:50.157+0000 7f73cdffb700 1 -- 172.21.15.29:0/3491273030 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7f73d40860a0 con 0x7f73d8101d20 2024-03-18T01:13:50.158 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:13:50.157+0000 7f73dcb58700 1 --2- 172.21.15.29:0/3491273030 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f73c4069210 0x7f73c406b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:13:50.158 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:13:50.157+0000 7f73dcb58700 1 --2- 172.21.15.29:0/3491273030 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f73c4069210 0x7f73c406b6c0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f73c8000c00 tx=0x7f73c8009e30 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:13:50.161 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:13:50.161+0000 7f73cdffb700 1 -- 172.21.15.29:0/3491273030 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175520 (secure 0 0 0) 0x7f73d4056f40 con 0x7f73d8101d20 2024-03-18T01:13:50.355 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:13:50.355+0000 7f73df5bd700 1 -- 172.21.15.29:0/3491273030 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7f73d8111230 con 0x7f73d8101d20 2024-03-18T01:13:50.356 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:13:50.355+0000 7f73cdffb700 1 -- 172.21.15.29:0/3491273030 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v10) v1 ==== 76+0+223 (secure 0 0 0) 0x7f73d4085850 con 0x7f73d8101d20 2024-03-18T01:13:50.356 INFO:teuthology.orchestra.run.smithi029.stdout:# minimal ceph.conf for 37f51182-e4c4-11ee-95c9-87774f69a715 2024-03-18T01:13:50.356 INFO:teuthology.orchestra.run.smithi029.stdout:[global] 2024-03-18T01:13:50.356 INFO:teuthology.orchestra.run.smithi029.stdout: fsid = 37f51182-e4c4-11ee-95c9-87774f69a715 2024-03-18T01:13:50.356 INFO:teuthology.orchestra.run.smithi029.stdout: mon_host = [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] 2024-03-18T01:13:50.358 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:13:50.357+0000 7f73df5bd700 1 -- 172.21.15.29:0/3491273030 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f73c4069210 msgr2=0x7f73c406b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:13:50.358 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:13:50.357+0000 7f73df5bd700 1 --2- 172.21.15.29:0/3491273030 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f73c4069210 0x7f73c406b6c0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f73c8000c00 tx=0x7f73c8009e30 comp rx=0 tx=0).stop 2024-03-18T01:13:50.358 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:13:50.357+0000 7f73df5bd700 1 -- 172.21.15.29:0/3491273030 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f73d8101d20 msgr2=0x7f73d8115bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:13:50.358 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:13:50.357+0000 7f73df5bd700 1 --2- 172.21.15.29:0/3491273030 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f73d8101d20 0x7f73d8115bb0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f73d400b790 tx=0x7f73d4009250 comp rx=0 tx=0).stop 2024-03-18T01:13:50.358 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:13:50.357+0000 7f73df5bd700 1 -- 172.21.15.29:0/3491273030 shutdown_connections 2024-03-18T01:13:50.358 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:13:50.357+0000 7f73df5bd700 1 --2- 172.21.15.29:0/3491273030 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f73c4069210 0x7f73c406b6c0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:13:50.358 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:13:50.357+0000 7f73df5bd700 1 --2- 172.21.15.29:0/3491273030 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f73d8101d20 0x7f73d8115bb0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:13:50.358 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:13:50.357+0000 7f73df5bd700 1 --2- 172.21.15.29:0/3491273030 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f73d81026c0 0x7f73d81160f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:13:50.358 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:13:50.357+0000 7f73df5bd700 1 -- 172.21.15.29:0/3491273030 >> 172.21.15.29:0/3491273030 conn(0x7f73d80f7870 msgr2=0x7f73d80f88b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:13:50.358 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:13:50.357+0000 7f73df5bd700 1 -- 172.21.15.29:0/3491273030 shutdown_connections 2024-03-18T01:13:50.358 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:13:50.357+0000 7f73df5bd700 1 -- 172.21.15.29:0/3491273030 wait complete. 2024-03-18T01:13:51.286 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:13:51 smithi029 ceph-mon[29331]: from='client.? 172.21.15.29:0/3491273030' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:13:51.286 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:13:51 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:13:51.286 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:13:51 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:13:51.286 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:13:51 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi029", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-03-18T01:13:51.286 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:13:51 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:13:51.309 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-03-18T01:13:51.309 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-03-18T01:13:51.309 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd of=/etc/ceph/ceph.conf 2024-03-18T01:13:51.338 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-03-18T01:13:51.338 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-18T01:13:51.406 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-03-18T01:13:51.407 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd of=/etc/ceph/ceph.conf 2024-03-18T01:13:51.436 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:13:51 smithi090 ceph-mon[36989]: from='client.? 172.21.15.29:0/3491273030' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:13:51.436 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:13:51 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:13:51.436 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:13:51 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:13:51.436 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:13:51 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi029", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-03-18T01:13:51.436 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:13:51 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:13:51.441 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-03-18T01:13:51.441 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-18T01:13:51.511 INFO:tasks.cephadm:Deploying OSDs... 2024-03-18T01:13:51.511 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-03-18T01:13:51.511 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/scratch_devs of=/dev/stdout 2024-03-18T01:13:51.526 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-18T01:13:51.526 DEBUG:teuthology.orchestra.run.smithi029:> stat /dev/vg_nvme/lv_1 2024-03-18T01:13:51.580 INFO:teuthology.orchestra.run.smithi029.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-18T01:13:51.581 INFO:teuthology.orchestra.run.smithi029.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-18T01:13:51.581 INFO:teuthology.orchestra.run.smithi029.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-03-18T01:13:51.581 INFO:teuthology.orchestra.run.smithi029.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-18T01:13:51.581 INFO:teuthology.orchestra.run.smithi029.stdout:Context: system_u:object_r:device_t:s0 2024-03-18T01:13:51.581 INFO:teuthology.orchestra.run.smithi029.stdout:Access: 2024-03-18 01:13:02.239882046 +0000 2024-03-18T01:13:51.581 INFO:teuthology.orchestra.run.smithi029.stdout:Modify: 2024-03-18 01:06:30.823907515 +0000 2024-03-18T01:13:51.581 INFO:teuthology.orchestra.run.smithi029.stdout:Change: 2024-03-18 01:06:30.823907515 +0000 2024-03-18T01:13:51.581 INFO:teuthology.orchestra.run.smithi029.stdout: Birth: 2024-03-18 01:06:30.823907515 +0000 2024-03-18T01:13:51.581 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-18T01:13:51.648 INFO:teuthology.orchestra.run.smithi029.stderr:1+0 records in 2024-03-18T01:13:51.648 INFO:teuthology.orchestra.run.smithi029.stderr:1+0 records out 2024-03-18T01:13:51.648 INFO:teuthology.orchestra.run.smithi029.stderr:512 bytes copied, 9.9027e-05 s, 5.2 MB/s 2024-03-18T01:13:51.649 DEBUG:teuthology.orchestra.run.smithi029:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-18T01:13:51.706 DEBUG:teuthology.orchestra.run.smithi029:> stat /dev/vg_nvme/lv_2 2024-03-18T01:13:51.760 INFO:teuthology.orchestra.run.smithi029.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-18T01:13:51.761 INFO:teuthology.orchestra.run.smithi029.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-18T01:13:51.761 INFO:teuthology.orchestra.run.smithi029.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-03-18T01:13:51.761 INFO:teuthology.orchestra.run.smithi029.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-18T01:13:51.761 INFO:teuthology.orchestra.run.smithi029.stdout:Context: system_u:object_r:device_t:s0 2024-03-18T01:13:51.761 INFO:teuthology.orchestra.run.smithi029.stdout:Access: 2024-03-18 01:13:02.239882046 +0000 2024-03-18T01:13:51.761 INFO:teuthology.orchestra.run.smithi029.stdout:Modify: 2024-03-18 01:06:31.157901822 +0000 2024-03-18T01:13:51.761 INFO:teuthology.orchestra.run.smithi029.stdout:Change: 2024-03-18 01:06:31.157901822 +0000 2024-03-18T01:13:51.761 INFO:teuthology.orchestra.run.smithi029.stdout: Birth: 2024-03-18 01:06:31.157901822 +0000 2024-03-18T01:13:51.761 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-18T01:13:51.826 INFO:teuthology.orchestra.run.smithi029.stderr:1+0 records in 2024-03-18T01:13:51.826 INFO:teuthology.orchestra.run.smithi029.stderr:1+0 records out 2024-03-18T01:13:51.826 INFO:teuthology.orchestra.run.smithi029.stderr:512 bytes copied, 9.9829e-05 s, 5.1 MB/s 2024-03-18T01:13:51.828 DEBUG:teuthology.orchestra.run.smithi029:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-18T01:13:51.883 DEBUG:teuthology.orchestra.run.smithi029:> stat /dev/vg_nvme/lv_3 2024-03-18T01:13:51.938 INFO:teuthology.orchestra.run.smithi029.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-18T01:13:51.938 INFO:teuthology.orchestra.run.smithi029.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-18T01:13:51.938 INFO:teuthology.orchestra.run.smithi029.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-03-18T01:13:51.938 INFO:teuthology.orchestra.run.smithi029.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-18T01:13:51.939 INFO:teuthology.orchestra.run.smithi029.stdout:Context: system_u:object_r:device_t:s0 2024-03-18T01:13:51.939 INFO:teuthology.orchestra.run.smithi029.stdout:Access: 2024-03-18 01:13:02.239882046 +0000 2024-03-18T01:13:51.939 INFO:teuthology.orchestra.run.smithi029.stdout:Modify: 2024-03-18 01:06:31.495896060 +0000 2024-03-18T01:13:51.939 INFO:teuthology.orchestra.run.smithi029.stdout:Change: 2024-03-18 01:06:31.495896060 +0000 2024-03-18T01:13:51.939 INFO:teuthology.orchestra.run.smithi029.stdout: Birth: 2024-03-18 01:06:31.495896060 +0000 2024-03-18T01:13:51.939 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-18T01:13:52.005 INFO:teuthology.orchestra.run.smithi029.stderr:1+0 records in 2024-03-18T01:13:52.005 INFO:teuthology.orchestra.run.smithi029.stderr:1+0 records out 2024-03-18T01:13:52.005 INFO:teuthology.orchestra.run.smithi029.stderr:512 bytes copied, 9.6784e-05 s, 5.3 MB/s 2024-03-18T01:13:52.006 DEBUG:teuthology.orchestra.run.smithi029:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-18T01:13:52.061 DEBUG:teuthology.orchestra.run.smithi029:> stat /dev/vg_nvme/lv_4 2024-03-18T01:13:52.116 INFO:teuthology.orchestra.run.smithi029.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-18T01:13:52.116 INFO:teuthology.orchestra.run.smithi029.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-18T01:13:52.116 INFO:teuthology.orchestra.run.smithi029.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-03-18T01:13:52.116 INFO:teuthology.orchestra.run.smithi029.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-18T01:13:52.116 INFO:teuthology.orchestra.run.smithi029.stdout:Context: system_u:object_r:device_t:s0 2024-03-18T01:13:52.116 INFO:teuthology.orchestra.run.smithi029.stdout:Access: 2024-03-18 01:13:02.240882029 +0000 2024-03-18T01:13:52.116 INFO:teuthology.orchestra.run.smithi029.stdout:Modify: 2024-03-18 01:06:31.836890248 +0000 2024-03-18T01:13:52.116 INFO:teuthology.orchestra.run.smithi029.stdout:Change: 2024-03-18 01:06:31.836890248 +0000 2024-03-18T01:13:52.116 INFO:teuthology.orchestra.run.smithi029.stdout: Birth: 2024-03-18 01:06:31.836890248 +0000 2024-03-18T01:13:52.116 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-18T01:13:52.180 INFO:teuthology.orchestra.run.smithi029.stderr:1+0 records in 2024-03-18T01:13:52.181 INFO:teuthology.orchestra.run.smithi029.stderr:1+0 records out 2024-03-18T01:13:52.181 INFO:teuthology.orchestra.run.smithi029.stderr:512 bytes copied, 9.7751e-05 s, 5.2 MB/s 2024-03-18T01:13:52.182 DEBUG:teuthology.orchestra.run.smithi029:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-18T01:13:52.237 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-03-18T01:13:52.237 DEBUG:teuthology.orchestra.run.smithi090:> dd if=/scratch_devs of=/dev/stdout 2024-03-18T01:13:52.256 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-18T01:13:52.256 DEBUG:teuthology.orchestra.run.smithi090:> stat /dev/vg_nvme/lv_1 2024-03-18T01:13:52.315 INFO:teuthology.orchestra.run.smithi090.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-18T01:13:52.315 INFO:teuthology.orchestra.run.smithi090.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-18T01:13:52.315 INFO:teuthology.orchestra.run.smithi090.stdout:Device: 5h/5d Inode: 668 Links: 1 2024-03-18T01:13:52.315 INFO:teuthology.orchestra.run.smithi090.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-18T01:13:52.315 INFO:teuthology.orchestra.run.smithi090.stdout:Context: system_u:object_r:device_t:s0 2024-03-18T01:13:52.315 INFO:teuthology.orchestra.run.smithi090.stdout:Access: 2024-03-18 01:13:37.144373154 +0000 2024-03-18T01:13:52.315 INFO:teuthology.orchestra.run.smithi090.stdout:Modify: 2024-03-18 01:06:24.027734033 +0000 2024-03-18T01:13:52.315 INFO:teuthology.orchestra.run.smithi090.stdout:Change: 2024-03-18 01:06:24.027734033 +0000 2024-03-18T01:13:52.315 INFO:teuthology.orchestra.run.smithi090.stdout: Birth: 2024-03-18 01:06:24.027734033 +0000 2024-03-18T01:13:52.316 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-18T01:13:52.319 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:13:52 smithi029 ceph-mon[29331]: Reconfiguring crash.smithi029 (monmap changed)... 2024-03-18T01:13:52.319 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:13:52 smithi029 ceph-mon[29331]: Reconfiguring daemon crash.smithi029 on smithi029 2024-03-18T01:13:52.319 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:13:52 smithi029 ceph-mon[29331]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T01:13:52.381 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:13:52 smithi090 ceph-mon[36989]: Reconfiguring crash.smithi029 (monmap changed)... 2024-03-18T01:13:52.381 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:13:52 smithi090 ceph-mon[36989]: Reconfiguring daemon crash.smithi029 on smithi029 2024-03-18T01:13:52.381 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:13:52 smithi090 ceph-mon[36989]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T01:13:52.384 INFO:teuthology.orchestra.run.smithi090.stderr:1+0 records in 2024-03-18T01:13:52.384 INFO:teuthology.orchestra.run.smithi090.stderr:1+0 records out 2024-03-18T01:13:52.384 INFO:teuthology.orchestra.run.smithi090.stderr:512 bytes copied, 0.000205502 s, 2.5 MB/s 2024-03-18T01:13:52.385 DEBUG:teuthology.orchestra.run.smithi090:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-18T01:13:52.442 DEBUG:teuthology.orchestra.run.smithi090:> stat /dev/vg_nvme/lv_2 2024-03-18T01:13:52.499 INFO:teuthology.orchestra.run.smithi090.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-18T01:13:52.500 INFO:teuthology.orchestra.run.smithi090.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-18T01:13:52.500 INFO:teuthology.orchestra.run.smithi090.stdout:Device: 5h/5d Inode: 687 Links: 1 2024-03-18T01:13:52.500 INFO:teuthology.orchestra.run.smithi090.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-18T01:13:52.500 INFO:teuthology.orchestra.run.smithi090.stdout:Context: system_u:object_r:device_t:s0 2024-03-18T01:13:52.500 INFO:teuthology.orchestra.run.smithi090.stdout:Access: 2024-03-18 01:13:37.144373154 +0000 2024-03-18T01:13:52.500 INFO:teuthology.orchestra.run.smithi090.stdout:Modify: 2024-03-18 01:06:24.362728522 +0000 2024-03-18T01:13:52.500 INFO:teuthology.orchestra.run.smithi090.stdout:Change: 2024-03-18 01:06:24.362728522 +0000 2024-03-18T01:13:52.500 INFO:teuthology.orchestra.run.smithi090.stdout: Birth: 2024-03-18 01:06:24.362728522 +0000 2024-03-18T01:13:52.500 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-18T01:13:52.568 INFO:teuthology.orchestra.run.smithi090.stderr:1+0 records in 2024-03-18T01:13:52.568 INFO:teuthology.orchestra.run.smithi090.stderr:1+0 records out 2024-03-18T01:13:52.568 INFO:teuthology.orchestra.run.smithi090.stderr:512 bytes copied, 0.000210924 s, 2.4 MB/s 2024-03-18T01:13:52.569 DEBUG:teuthology.orchestra.run.smithi090:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-18T01:13:52.626 DEBUG:teuthology.orchestra.run.smithi090:> stat /dev/vg_nvme/lv_3 2024-03-18T01:13:52.682 INFO:teuthology.orchestra.run.smithi090.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-18T01:13:52.682 INFO:teuthology.orchestra.run.smithi090.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-18T01:13:52.682 INFO:teuthology.orchestra.run.smithi090.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-03-18T01:13:52.683 INFO:teuthology.orchestra.run.smithi090.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-18T01:13:52.683 INFO:teuthology.orchestra.run.smithi090.stdout:Context: system_u:object_r:device_t:s0 2024-03-18T01:13:52.683 INFO:teuthology.orchestra.run.smithi090.stdout:Access: 2024-03-18 01:13:37.145373138 +0000 2024-03-18T01:13:52.683 INFO:teuthology.orchestra.run.smithi090.stdout:Modify: 2024-03-18 01:06:24.714722731 +0000 2024-03-18T01:13:52.683 INFO:teuthology.orchestra.run.smithi090.stdout:Change: 2024-03-18 01:06:24.714722731 +0000 2024-03-18T01:13:52.683 INFO:teuthology.orchestra.run.smithi090.stdout: Birth: 2024-03-18 01:06:24.714722731 +0000 2024-03-18T01:13:52.683 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-18T01:13:52.749 INFO:teuthology.orchestra.run.smithi090.stderr:1+0 records in 2024-03-18T01:13:52.749 INFO:teuthology.orchestra.run.smithi090.stderr:1+0 records out 2024-03-18T01:13:52.749 INFO:teuthology.orchestra.run.smithi090.stderr:512 bytes copied, 0.000210977 s, 2.4 MB/s 2024-03-18T01:13:52.750 DEBUG:teuthology.orchestra.run.smithi090:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-18T01:13:52.807 DEBUG:teuthology.orchestra.run.smithi090:> stat /dev/vg_nvme/lv_4 2024-03-18T01:13:52.862 INFO:teuthology.orchestra.run.smithi090.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-18T01:13:52.863 INFO:teuthology.orchestra.run.smithi090.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-18T01:13:52.863 INFO:teuthology.orchestra.run.smithi090.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-03-18T01:13:52.863 INFO:teuthology.orchestra.run.smithi090.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-18T01:13:52.863 INFO:teuthology.orchestra.run.smithi090.stdout:Context: system_u:object_r:device_t:s0 2024-03-18T01:13:52.863 INFO:teuthology.orchestra.run.smithi090.stdout:Access: 2024-03-18 01:13:37.145373138 +0000 2024-03-18T01:13:52.863 INFO:teuthology.orchestra.run.smithi090.stdout:Modify: 2024-03-18 01:06:25.044717303 +0000 2024-03-18T01:13:52.863 INFO:teuthology.orchestra.run.smithi090.stdout:Change: 2024-03-18 01:06:25.044717303 +0000 2024-03-18T01:13:52.863 INFO:teuthology.orchestra.run.smithi090.stdout: Birth: 2024-03-18 01:06:25.044717303 +0000 2024-03-18T01:13:52.863 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-18T01:13:52.928 INFO:teuthology.orchestra.run.smithi090.stderr:1+0 records in 2024-03-18T01:13:52.928 INFO:teuthology.orchestra.run.smithi090.stderr:1+0 records out 2024-03-18T01:13:52.928 INFO:teuthology.orchestra.run.smithi090.stderr:512 bytes copied, 0.000197005 s, 2.6 MB/s 2024-03-18T01:13:52.929 DEBUG:teuthology.orchestra.run.smithi090:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-18T01:13:52.985 INFO:tasks.cephadm:Deploying osd.0 on smithi029 with /dev/vg_nvme/lv_4... 2024-03-18T01:13:52.985 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 37f51182-e4c4-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-18T01:13:53.883 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:13:53 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:13:53.884 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:13:53 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:13:53.884 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:13:53 smithi029 ceph-mon[29331]: Reconfiguring alertmanager.smithi029 (dependencies changed)... 2024-03-18T01:13:53.884 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:13:53 smithi029 ceph-mon[29331]: Reconfiguring daemon alertmanager.smithi029 on smithi029 2024-03-18T01:13:53.884 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:13:53 smithi029 ceph-mon[29331]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T01:13:54.026 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:13:53 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:13:54.026 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:13:53 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:13:54.026 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:13:53 smithi090 ceph-mon[36989]: Reconfiguring alertmanager.smithi029 (dependencies changed)... 2024-03-18T01:13:54.026 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:13:53 smithi090 ceph-mon[36989]: Reconfiguring daemon alertmanager.smithi029 on smithi029 2024-03-18T01:13:54.027 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:13:53 smithi090 ceph-mon[36989]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T01:13:55.137 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/37f51182-e4c4-11ee-95c9-87774f69a715/mon.smithi029/config 2024-03-18T01:13:56.258 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:13:56 smithi029 ceph-mon[29331]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T01:13:56.258 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:13:56 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-18T01:13:56.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:13:56 smithi090 ceph-mon[36989]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T01:13:56.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:13:56 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-18T01:13:57.740 INFO:teuthology.orchestra.run.smithi029.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-18T01:13:57.741 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-18T01:13:57.741 INFO:teuthology.orchestra.run.smithi029.stderr: stderr: 10+0 records in 2024-03-18T01:13:57.741 INFO:teuthology.orchestra.run.smithi029.stderr:10+0 records out 2024-03-18T01:13:57.741 INFO:teuthology.orchestra.run.smithi029.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.00838963 s, 1.2 GB/s 2024-03-18T01:13:57.741 INFO:teuthology.orchestra.run.smithi029.stderr: stderr: 2024-03-18T01:13:57.741 INFO:teuthology.orchestra.run.smithi029.stderr:--> Zapping successful for: 2024-03-18T01:13:58.319 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:13:58 smithi029 ceph-mon[29331]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T01:13:58.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:13:58 smithi090 ceph-mon[36989]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T01:13:58.739 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 37f51182-e4c4-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi029:vg_nvme/lv_4 2024-03-18T01:13:59.118 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/37f51182-e4c4-11ee-95c9-87774f69a715/mon.smithi029/config 2024-03-18T01:14:00.319 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:14:00 smithi029 ceph-mon[29331]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T01:14:00.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:14:00 smithi090 ceph-mon[36989]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T01:14:01.776 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:14:01 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:14:01.776 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:14:01 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:14:01.776 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:14:01 smithi090 ceph-mon[36989]: Reconfiguring grafana.smithi029 (dependencies changed)... 2024-03-18T01:14:01.776 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:14:01 smithi090 ceph-mon[36989]: Reconfiguring daemon grafana.smithi029 on smithi029 2024-03-18T01:14:01.776 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:14:01 smithi090 ceph-mon[36989]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T01:14:01.819 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:14:01 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:14:01.819 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:14:01 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:14:01.819 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:14:01 smithi029 ceph-mon[29331]: Reconfiguring grafana.smithi029 (dependencies changed)... 2024-03-18T01:14:01.819 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:14:01 smithi029 ceph-mon[29331]: Reconfiguring daemon grafana.smithi029 on smithi029 2024-03-18T01:14:01.819 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:14:01 smithi029 ceph-mon[29331]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T01:14:02.308 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:14:02.268+0000 7f92e2488700 1 -- 172.21.15.29:0/1796364547 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f92dc0fbcd0 msgr2=0x7f92dc0fc0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:14:02.308 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:14:02.268+0000 7f92e2488700 1 --2- 172.21.15.29:0/1796364547 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f92dc0fbcd0 0x7f92dc0fc0e0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f92cc004940 tx=0x7f92cc009e30 comp rx=0 tx=0).stop 2024-03-18T01:14:02.308 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:14:02.268+0000 7f92e2488700 1 -- 172.21.15.29:0/1796364547 shutdown_connections 2024-03-18T01:14:02.308 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:14:02.268+0000 7f92e2488700 1 --2- 172.21.15.29:0/1796364547 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f92dc0fcf80 0x7f92dc0fd3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:14:02.308 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:14:02.268+0000 7f92e2488700 1 --2- 172.21.15.29:0/1796364547 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f92dc0fbcd0 0x7f92dc0fc0e0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:14:02.308 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:14:02.268+0000 7f92e2488700 1 -- 172.21.15.29:0/1796364547 >> 172.21.15.29:0/1796364547 conn(0x7f92dc0f7450 msgr2=0x7f92dc0f9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:14:02.308 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:14:02.269+0000 7f92e2488700 1 -- 172.21.15.29:0/1796364547 shutdown_connections 2024-03-18T01:14:02.308 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:14:02.269+0000 7f92e2488700 1 -- 172.21.15.29:0/1796364547 wait complete. 2024-03-18T01:14:02.309 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:14:02.269+0000 7f92e2488700 1 Processor -- start 2024-03-18T01:14:02.309 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:14:02.269+0000 7f92e2488700 1 -- start start 2024-03-18T01:14:02.309 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:14:02.269+0000 7f92e2488700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f92dc0fbcd0 0x7f92dc066cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:14:02.309 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:14:02.269+0000 7f92e2488700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f92dc0fcf80 0x7f92dc114b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:14:02.309 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:14:02.269+0000 7f92e2488700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92dc10e530 con 0x7f92dc0fbcd0 2024-03-18T01:14:02.309 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:14:02.269+0000 7f92e2488700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92dc115090 con 0x7f92dc0fcf80 2024-03-18T01:14:02.309 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:14:02.270+0000 7f92dbfff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f92dc0fbcd0 0x7f92dc066cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:14:02.309 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:14:02.270+0000 7f92dbfff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f92dc0fbcd0 0x7f92dc066cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56134/0 (socket says 172.21.15.29:56134) 2024-03-18T01:14:02.309 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:14:02.270+0000 7f92dbfff700 1 -- 172.21.15.29:0/3826897427 learned_addr learned my addr 172.21.15.29:0/3826897427 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-18T01:14:02.309 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:14:02.270+0000 7f92db7fe700 1 --2- 172.21.15.29:0/3826897427 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f92dc0fcf80 0x7f92dc114b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:14:02.309 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:14:02.270+0000 7f92dbfff700 1 -- 172.21.15.29:0/3826897427 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f92dc0fcf80 msgr2=0x7f92dc114b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:14:02.309 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:14:02.270+0000 7f92dbfff700 1 --2- 172.21.15.29:0/3826897427 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f92dc0fcf80 0x7f92dc114b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:14:02.309 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:14:02.270+0000 7f92dbfff700 1 -- 172.21.15.29:0/3826897427 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92cc005040 con 0x7f92dc0fbcd0 2024-03-18T01:14:02.309 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:14:02.270+0000 7f92dbfff700 1 --2- 172.21.15.29:0/3826897427 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f92dc0fbcd0 0x7f92dc066cd0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f92cc009180 tx=0x7f92cc009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:14:02.310 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:14:02.270+0000 7f92d8ff9700 1 -- 172.21.15.29:0/3826897427 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f92cc00e040 con 0x7f92dc0fbcd0 2024-03-18T01:14:02.310 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:14:02.270+0000 7f92d8ff9700 1 -- 172.21.15.29:0/3826897427 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f92cc00bbf0 con 0x7f92dc0fbcd0 2024-03-18T01:14:02.310 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:14:02.270+0000 7f92e2488700 1 -- 172.21.15.29:0/3826897427 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92dc115230 con 0x7f92dc0fbcd0 2024-03-18T01:14:02.310 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:14:02.270+0000 7f92d8ff9700 1 -- 172.21.15.29:0/3826897427 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f92cc01c630 con 0x7f92dc0fbcd0 2024-03-18T01:14:02.310 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:14:02.270+0000 7f92e2488700 1 -- 172.21.15.29:0/3826897427 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92dc1157a0 con 0x7f92dc0fbcd0 2024-03-18T01:14:02.310 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:14:02.271+0000 7f92e2488700 1 -- 172.21.15.29:0/3826897427 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92dc04c710 con 0x7f92dc0fbcd0 2024-03-18T01:14:02.310 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:14:02.272+0000 7f92d8ff9700 1 -- 172.21.15.29:0/3826897427 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 18) v1 ==== 86565+0+0 (secure 0 0 0) 0x7f92cc00bd60 con 0x7f92dc0fbcd0 2024-03-18T01:14:02.310 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:14:02.272+0000 7f92d8ff9700 1 --2- 172.21.15.29:0/3826897427 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f92c4069210 0x7f92c406b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:14:02.310 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:14:02.272+0000 7f92d8ff9700 1 -- 172.21.15.29:0/3826897427 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7f92cc0873a0 con 0x7f92dc0fbcd0 2024-03-18T01:14:02.310 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:14:02.272+0000 7f92db7fe700 1 --2- 172.21.15.29:0/3826897427 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f92c4069210 0x7f92c406b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:14:02.310 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:14:02.272+0000 7f92db7fe700 1 --2- 172.21.15.29:0/3826897427 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f92c4069210 0x7f92c406b6c0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f92d0004f40 tx=0x7f92d000b040 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:14:02.310 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:14:02.277+0000 7f92d8ff9700 1 -- 172.21.15.29:0/3826897427 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175520 (secure 0 0 0) 0x7f92cc0582f0 con 0x7f92dc0fbcd0 2024-03-18T01:14:02.484 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:14:02.483+0000 7f92e2488700 1 -- 172.21.15.29:0/3826897427 --> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi029:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7f92dc1012e0 con 0x7f92c4069210 2024-03-18T01:14:02.776 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:14:02 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T01:14:02.776 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:14:02 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T01:14:02.776 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:14:02 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:14:02.819 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:14:02 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T01:14:02.819 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:14:02 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T01:14:02.819 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:14:02 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:14:03.776 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:14:03 smithi090 ceph-mon[36989]: from='client.14246 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi029:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T01:14:03.776 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:14:03 smithi090 ceph-mon[36989]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T01:14:03.819 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:14:03 smithi029 ceph-mon[29331]: from='client.14246 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi029:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T01:14:03.819 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:14:03 smithi029 ceph-mon[29331]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T01:14:06.294 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:14:06 smithi029 ceph-mon[29331]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T01:14:06.525 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:14:06 smithi090 ceph-mon[36989]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T01:14:08.026 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:14:07 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:14:08.026 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:14:07 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:14:08.026 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:14:07 smithi090 ceph-mon[36989]: Reconfiguring prometheus.smithi029 (dependencies changed)... 2024-03-18T01:14:08.026 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:14:07 smithi090 ceph-mon[36989]: Reconfiguring daemon prometheus.smithi029 on smithi029 2024-03-18T01:14:08.026 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:14:07 smithi090 ceph-mon[36989]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T01:14:08.069 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:14:07 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:14:08.069 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:14:07 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:14:08.069 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:14:07 smithi029 ceph-mon[29331]: Reconfiguring prometheus.smithi029 (dependencies changed)... 2024-03-18T01:14:08.069 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:14:07 smithi029 ceph-mon[29331]: Reconfiguring daemon prometheus.smithi029 on smithi029 2024-03-18T01:14:08.069 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:14:07 smithi029 ceph-mon[29331]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T01:14:08.819 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:14:08 smithi029 ceph-mon[29331]: from='client.? 172.21.15.29:0/905336501' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "decb820d-5c65-4317-b3f0-46522f1d1326"}]: dispatch 2024-03-18T01:14:08.819 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:14:08 smithi029 ceph-mon[29331]: from='client.? 172.21.15.29:0/905336501' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "decb820d-5c65-4317-b3f0-46522f1d1326"}]': finished 2024-03-18T01:14:08.819 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:14:08 smithi029 ceph-mon[29331]: osdmap e6: 1 total, 0 up, 1 in 2024-03-18T01:14:08.819 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:14:08 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-18T01:14:08.819 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:14:08 smithi029 ceph-mon[29331]: from='client.? 172.21.15.29:0/3127506203' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T01:14:09.026 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:14:08 smithi090 ceph-mon[36989]: from='client.? 172.21.15.29:0/905336501' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "decb820d-5c65-4317-b3f0-46522f1d1326"}]: dispatch 2024-03-18T01:14:09.026 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:14:08 smithi090 ceph-mon[36989]: from='client.? 172.21.15.29:0/905336501' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "decb820d-5c65-4317-b3f0-46522f1d1326"}]': finished 2024-03-18T01:14:09.026 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:14:08 smithi090 ceph-mon[36989]: osdmap e6: 1 total, 0 up, 1 in 2024-03-18T01:14:09.026 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:14:08 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-18T01:14:09.026 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:14:08 smithi090 ceph-mon[36989]: from='client.? 172.21.15.29:0/3127506203' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T01:14:10.025 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:14:09 smithi090 ceph-mon[36989]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T01:14:10.069 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:14:09 smithi029 ceph-mon[29331]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T01:14:11.319 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:14:11 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-18T01:14:11.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:14:11 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-18T01:14:12.319 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:14:12 smithi029 ceph-mon[29331]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T01:14:12.525 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:14:12 smithi090 ceph-mon[36989]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T01:14:14.319 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:14:14 smithi029 ceph-mon[29331]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T01:14:14.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:14:14 smithi090 ceph-mon[36989]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T01:14:16.386 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:14:16 smithi029 ceph-mon[29331]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T01:14:16.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:14:16 smithi090 ceph-mon[36989]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T01:14:18.276 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:14:18 smithi029 ceph-mon[29331]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T01:14:18.516 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:14:18 smithi090 ceph-mon[36989]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T01:14:19.231 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:14:19 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:14:19.231 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:14:19 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:14:19.232 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:14:19 smithi029 ceph-mon[29331]: Reconfiguring crash.smithi090 (monmap changed)... 2024-03-18T01:14:19.232 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:14:19 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi090", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-03-18T01:14:19.232 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:14:19 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:14:19.232 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:14:19 smithi029 ceph-mon[29331]: Reconfiguring daemon crash.smithi090 on smithi090 2024-03-18T01:14:19.374 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:14:19 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:14:19.374 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:14:19 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:14:19.375 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:14:19 smithi090 ceph-mon[36989]: Reconfiguring crash.smithi090 (monmap changed)... 2024-03-18T01:14:19.375 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:14:19 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi090", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-03-18T01:14:19.375 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:14:19 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:14:19.375 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:14:19 smithi090 ceph-mon[36989]: Reconfiguring daemon crash.smithi090 on smithi090 2024-03-18T01:14:20.274 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:14:20 smithi029 ceph-mon[29331]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T01:14:20.274 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:14:20 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:14:20.274 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:14:20 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:14:20.275 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:14:20 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi090.pnqedy", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-18T01:14:20.275 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:14:20 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-18T01:14:20.275 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:14:20 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:14:20.275 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:14:20 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-18T01:14:20.275 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:14:20 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:14:20.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:14:20 smithi090 ceph-mon[36989]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T01:14:20.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:14:20 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:14:20.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:14:20 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:14:20.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:14:20 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi090.pnqedy", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-18T01:14:20.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:14:20 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-18T01:14:20.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:14:20 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:14:20.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:14:20 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-18T01:14:20.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:14:20 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:14:21.292 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:14:21 smithi029 ceph-mon[29331]: Reconfiguring mgr.smithi090.pnqedy (monmap changed)... 2024-03-18T01:14:21.293 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:14:21 smithi029 ceph-mon[29331]: Reconfiguring daemon mgr.smithi090.pnqedy on smithi090 2024-03-18T01:14:21.293 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:14:21 smithi029 ceph-mon[29331]: Deploying daemon osd.0 on smithi029 2024-03-18T01:14:21.293 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:14:21 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:14:21.293 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:14:21 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:14:21.293 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:14:21 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-18T01:14:21.293 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:14:21 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-18T01:14:21.293 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:14:21 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:14:21.445 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:14:21 smithi090 ceph-mon[36989]: Reconfiguring mgr.smithi090.pnqedy (monmap changed)... 2024-03-18T01:14:21.445 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:14:21 smithi090 ceph-mon[36989]: Reconfiguring daemon mgr.smithi090.pnqedy on smithi090 2024-03-18T01:14:21.445 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:14:21 smithi090 ceph-mon[36989]: Deploying daemon osd.0 on smithi029 2024-03-18T01:14:21.445 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:14:21 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:14:21.445 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:14:21 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:14:21.445 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:14:21 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-18T01:14:21.445 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:14:21 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-18T01:14:21.446 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:14:21 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:14:22.371 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:14:22 smithi029 ceph-mon[29331]: Reconfiguring mon.smithi090 (monmap changed)... 2024-03-18T01:14:22.371 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:14:22 smithi029 ceph-mon[29331]: Reconfiguring daemon mon.smithi090 on smithi090 2024-03-18T01:14:22.371 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:14:22 smithi029 ceph-mon[29331]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T01:14:22.371 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:14:22 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:14:22.371 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:14:22 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:14:22.371 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:14:22 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-18T01:14:22.371 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:14:22 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi029.front.sepia.ceph.com:9093"}]: dispatch 2024-03-18T01:14:22.371 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:14:22 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:14:22.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:14:22 smithi090 ceph-mon[36989]: Reconfiguring mon.smithi090 (monmap changed)... 2024-03-18T01:14:22.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:14:22 smithi090 ceph-mon[36989]: Reconfiguring daemon mon.smithi090 on smithi090 2024-03-18T01:14:22.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:14:22 smithi090 ceph-mon[36989]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T01:14:22.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:14:22 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:14:22.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:14:22 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:14:22.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:14:22 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-18T01:14:22.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:14:22 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi029.front.sepia.ceph.com:9093"}]: dispatch 2024-03-18T01:14:22.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:14:22 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:14:23.319 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:14:23 smithi029 ceph-mon[29331]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-18T01:14:23.320 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:14:23 smithi029 ceph-mon[29331]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi029.front.sepia.ceph.com:9093"}]: dispatch 2024-03-18T01:14:23.320 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:14:23 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-18T01:14:23.320 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:14:23 smithi029 ceph-mon[29331]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-18T01:14:23.320 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:14:23 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi029.front.sepia.ceph.com:3000"}]: dispatch 2024-03-18T01:14:23.320 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:14:23 smithi029 ceph-mon[29331]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi029.front.sepia.ceph.com:3000"}]: dispatch 2024-03-18T01:14:23.320 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:14:23 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:14:23.320 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:14:23 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-18T01:14:23.320 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:14:23 smithi029 ceph-mon[29331]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-18T01:14:23.320 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:14:23 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi029.front.sepia.ceph.com:9095"}]: dispatch 2024-03-18T01:14:23.320 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:14:23 smithi029 ceph-mon[29331]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi029.front.sepia.ceph.com:9095"}]: dispatch 2024-03-18T01:14:23.320 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:14:23 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:14:23.320 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:14:23 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T01:14:23.370 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:14:23 smithi090 ceph-mon[36989]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-18T01:14:23.370 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:14:23 smithi090 ceph-mon[36989]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi029.front.sepia.ceph.com:9093"}]: dispatch 2024-03-18T01:14:23.370 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:14:23 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-18T01:14:23.370 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:14:23 smithi090 ceph-mon[36989]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-18T01:14:23.370 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:14:23 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi029.front.sepia.ceph.com:3000"}]: dispatch 2024-03-18T01:14:23.370 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:14:23 smithi090 ceph-mon[36989]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi029.front.sepia.ceph.com:3000"}]: dispatch 2024-03-18T01:14:23.370 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:14:23 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:14:23.370 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:14:23 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-18T01:14:23.370 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:14:23 smithi090 ceph-mon[36989]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-18T01:14:23.370 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:14:23 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi029.front.sepia.ceph.com:9095"}]: dispatch 2024-03-18T01:14:23.370 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:14:23 smithi090 ceph-mon[36989]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi029.front.sepia.ceph.com:9095"}]: dispatch 2024-03-18T01:14:23.371 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:14:23 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:14:23.371 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:14:23 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T01:14:24.298 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:14:24 smithi029 ceph-mon[29331]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T01:14:24.299 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:14:24 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:14:24.299 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:14:24 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:14:24.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:14:24 smithi090 ceph-mon[36989]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T01:14:24.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:14:24 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:14:24.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:14:24 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:14:26.339 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:14:26 smithi029 ceph-mon[29331]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T01:14:26.339 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:14:26 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:14:26.339 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:14:26 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:14:26.339 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:14:26 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-18T01:14:26.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:14:26 smithi090 ceph-mon[36989]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T01:14:26.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:14:26 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:14:26.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:14:26 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:14:26.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:14:26 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-18T01:14:28.445 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:14:28 smithi029 ceph-mon[29331]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T01:14:28.525 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:14:28 smithi090 ceph-mon[36989]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T01:14:30.271 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:14:30 smithi029 ceph-mon[29331]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T01:14:30.272 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:14:30 smithi029 ceph-mon[29331]: from='osd.0 [v2:172.21.15.29:6802/895619893,v1:172.21.15.29:6803/895619893]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-18T01:14:30.312 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:14:30.311+0000 7f92d8ff9700 1 -- 172.21.15.29:0/3826897427 <== mgr.14208 v2:172.21.15.29:6800/1641454714 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f92dc1012e0 con 0x7f92c4069210 2024-03-18T01:14:30.312 INFO:teuthology.orchestra.run.smithi029.stdout:Created osd(s) 0 on host 'smithi029' 2024-03-18T01:14:30.314 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:14:30.313+0000 7f92e2488700 1 -- 172.21.15.29:0/3826897427 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f92c4069210 msgr2=0x7f92c406b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:14:30.314 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:14:30.313+0000 7f92e2488700 1 --2- 172.21.15.29:0/3826897427 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f92c4069210 0x7f92c406b6c0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f92d0004f40 tx=0x7f92d000b040 comp rx=0 tx=0).stop 2024-03-18T01:14:30.314 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:14:30.313+0000 7f92e2488700 1 -- 172.21.15.29:0/3826897427 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f92dc0fbcd0 msgr2=0x7f92dc066cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:14:30.314 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:14:30.313+0000 7f92e2488700 1 --2- 172.21.15.29:0/3826897427 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f92dc0fbcd0 0x7f92dc066cd0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f92cc009180 tx=0x7f92cc009e30 comp rx=0 tx=0).stop 2024-03-18T01:14:30.314 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:14:30.313+0000 7f92e2488700 1 -- 172.21.15.29:0/3826897427 shutdown_connections 2024-03-18T01:14:30.314 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:14:30.313+0000 7f92e2488700 1 --2- 172.21.15.29:0/3826897427 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f92c4069210 0x7f92c406b6c0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:14:30.315 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:14:30.313+0000 7f92e2488700 1 --2- 172.21.15.29:0/3826897427 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f92dc0fbcd0 0x7f92dc066cd0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:14:30.315 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:14:30.313+0000 7f92e2488700 1 --2- 172.21.15.29:0/3826897427 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f92dc0fcf80 0x7f92dc114b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:14:30.315 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:14:30.313+0000 7f92e2488700 1 -- 172.21.15.29:0/3826897427 >> 172.21.15.29:0/3826897427 conn(0x7f92dc0f7450 msgr2=0x7f92dc0f8b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:14:30.315 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:14:30.314+0000 7f92e2488700 1 -- 172.21.15.29:0/3826897427 shutdown_connections 2024-03-18T01:14:30.315 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:14:30.314+0000 7f92e2488700 1 -- 172.21.15.29:0/3826897427 wait complete. 2024-03-18T01:14:30.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:14:30 smithi090 ceph-mon[36989]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T01:14:30.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:14:30 smithi090 ceph-mon[36989]: from='osd.0 [v2:172.21.15.29:6802/895619893,v1:172.21.15.29:6803/895619893]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-18T01:14:30.975 DEBUG:teuthology.orchestra.run.smithi029:osd.0> sudo journalctl -f -n 0 -u ceph-37f51182-e4c4-11ee-95c9-87774f69a715@osd.0.service 2024-03-18T01:14:30.978 INFO:tasks.cephadm:Deploying osd.1 on smithi029 with /dev/vg_nvme/lv_3... 2024-03-18T01:14:30.978 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 37f51182-e4c4-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-18T01:14:31.296 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/37f51182-e4c4-11ee-95c9-87774f69a715/mon.smithi029/config 2024-03-18T01:14:31.319 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:14:31 smithi029 ceph-mon[29331]: from='osd.0 [v2:172.21.15.29:6802/895619893,v1:172.21.15.29:6803/895619893]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-18T01:14:31.319 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:14:31 smithi029 ceph-mon[29331]: osdmap e7: 1 total, 0 up, 1 in 2024-03-18T01:14:31.319 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:14:31 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-18T01:14:31.319 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:14:31 smithi029 ceph-mon[29331]: from='osd.0 [v2:172.21.15.29:6802/895619893,v1:172.21.15.29:6803/895619893]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]: dispatch 2024-03-18T01:14:31.319 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:14:31 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:14:31.319 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:14:31 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:14:31.319 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:14:31 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:14:31.319 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:14:31 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:14:31.319 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:14:31 smithi029 ceph-mon[29331]: from='osd.0 [v2:172.21.15.29:6802/895619893,v1:172.21.15.29:6803/895619893]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]': finished 2024-03-18T01:14:31.319 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:14:31 smithi029 ceph-mon[29331]: osdmap e8: 1 total, 0 up, 1 in 2024-03-18T01:14:31.319 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:14:31 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-18T01:14:31.319 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:14:31 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-18T01:14:31.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:14:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:14:31.151+0000 7f7850feb700 -1 osd.0 0 waiting for initial osdmap 2024-03-18T01:14:31.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:14:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:14:31.155+0000 7f784c5e7700 -1 osd.0 8 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-18T01:14:31.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:14:31 smithi090 ceph-mon[36989]: from='osd.0 [v2:172.21.15.29:6802/895619893,v1:172.21.15.29:6803/895619893]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-18T01:14:31.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:14:31 smithi090 ceph-mon[36989]: osdmap e7: 1 total, 0 up, 1 in 2024-03-18T01:14:31.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:14:31 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-18T01:14:31.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:14:31 smithi090 ceph-mon[36989]: from='osd.0 [v2:172.21.15.29:6802/895619893,v1:172.21.15.29:6803/895619893]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]: dispatch 2024-03-18T01:14:31.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:14:31 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:14:31.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:14:31 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:14:31.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:14:31 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:14:31.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:14:31 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:14:31.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:14:31 smithi090 ceph-mon[36989]: from='osd.0 [v2:172.21.15.29:6802/895619893,v1:172.21.15.29:6803/895619893]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]': finished 2024-03-18T01:14:31.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:14:31 smithi090 ceph-mon[36989]: osdmap e8: 1 total, 0 up, 1 in 2024-03-18T01:14:31.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:14:31 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-18T01:14:31.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:14:31 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-18T01:14:32.411 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:14:32 smithi029 ceph-mon[29331]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T01:14:32.411 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:14:32 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-18T01:14:32.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:14:32 smithi090 ceph-mon[36989]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T01:14:32.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:14:32 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-18T01:14:33.276 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:14:33 smithi029 ceph-mon[29331]: purged_snaps scrub starts 2024-03-18T01:14:33.276 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:14:33 smithi029 ceph-mon[29331]: purged_snaps scrub ok 2024-03-18T01:14:33.277 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:14:33 smithi029 ceph-mon[29331]: osd.0 [v2:172.21.15.29:6802/895619893,v1:172.21.15.29:6803/895619893] boot 2024-03-18T01:14:33.277 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:14:33 smithi029 ceph-mon[29331]: osdmap e9: 1 total, 1 up, 1 in 2024-03-18T01:14:33.277 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:14:33 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-18T01:14:33.288 INFO:teuthology.orchestra.run.smithi029.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-18T01:14:33.288 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-18T01:14:33.288 INFO:teuthology.orchestra.run.smithi029.stderr: stderr: 10+0 records in 2024-03-18T01:14:33.288 INFO:teuthology.orchestra.run.smithi029.stderr:10+0 records out 2024-03-18T01:14:33.288 INFO:teuthology.orchestra.run.smithi029.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0100022 s, 1.0 GB/s 2024-03-18T01:14:33.288 INFO:teuthology.orchestra.run.smithi029.stderr:--> Zapping successful for: 2024-03-18T01:14:33.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:14:33 smithi090 ceph-mon[36989]: purged_snaps scrub starts 2024-03-18T01:14:33.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:14:33 smithi090 ceph-mon[36989]: purged_snaps scrub ok 2024-03-18T01:14:33.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:14:33 smithi090 ceph-mon[36989]: osd.0 [v2:172.21.15.29:6802/895619893,v1:172.21.15.29:6803/895619893] boot 2024-03-18T01:14:33.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:14:33 smithi090 ceph-mon[36989]: osdmap e9: 1 total, 1 up, 1 in 2024-03-18T01:14:33.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:14:33 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-18T01:14:34.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:14:34 smithi090 ceph-mon[36989]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T01:14:34.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:14:34 smithi090 ceph-mon[36989]: osdmap e10: 1 total, 1 up, 1 in 2024-03-18T01:14:34.544 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:14:34 smithi029 ceph-mon[29331]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T01:14:34.544 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:14:34 smithi029 ceph-mon[29331]: osdmap e10: 1 total, 1 up, 1 in 2024-03-18T01:14:35.527 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 37f51182-e4c4-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi029:vg_nvme/lv_3 2024-03-18T01:14:36.366 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:14:36 smithi029 ceph-mon[29331]: pgmap v40: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-03-18T01:14:36.459 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/37f51182-e4c4-11ee-95c9-87774f69a715/mon.smithi029/config 2024-03-18T01:14:36.525 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:14:36 smithi090 ceph-mon[36989]: pgmap v40: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-03-18T01:14:37.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:14:37 smithi029 ceph-mon[29331]: Detected new or changed devices on smithi029 2024-03-18T01:14:37.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:14:37 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:14:37.570 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:14:37 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:14:37.570 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:14:37 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-18T01:14:37.570 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:14:37 smithi029 ceph-mon[29331]: Adjusting osd_memory_target on smithi029 to 12869M 2024-03-18T01:14:37.570 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:14:37 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:14:37.570 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:14:37 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:14:37.570 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:14:37 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T01:14:37.570 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:14:37 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:14:37.570 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:14:37 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T01:14:37.570 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:14:37 smithi029 ceph-mon[29331]: pgmap v41: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-03-18T01:14:37.732 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:14:37 smithi090 ceph-mon[36989]: Detected new or changed devices on smithi029 2024-03-18T01:14:37.732 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:14:37 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:14:37.732 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:14:37 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:14:37.732 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:14:37 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-18T01:14:37.732 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:14:37 smithi090 ceph-mon[36989]: Adjusting osd_memory_target on smithi029 to 12869M 2024-03-18T01:14:37.732 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:14:37 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:14:37.732 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:14:37 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:14:37.732 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:14:37 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T01:14:37.732 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:14:37 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:14:37.732 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:14:37 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T01:14:37.732 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:14:37 smithi090 ceph-mon[36989]: pgmap v41: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-03-18T01:14:38.170 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:14:38.169+0000 7fb02820c700 1 -- 172.21.15.29:0/1979686630 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb0200fed90 msgr2=0x7fb0200faed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:14:38.170 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:14:38.169+0000 7fb02820c700 1 --2- 172.21.15.29:0/1979686630 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb0200fed90 0x7fb0200faed0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fb014004940 tx=0x7fb014009e30 comp rx=0 tx=0).stop 2024-03-18T01:14:38.170 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:14:38.170+0000 7fb02820c700 1 -- 172.21.15.29:0/1979686630 shutdown_connections 2024-03-18T01:14:38.170 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:14:38.170+0000 7fb02820c700 1 --2- 172.21.15.29:0/1979686630 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb0200fed90 0x7fb0200faed0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:14:38.171 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:14:38.170+0000 7fb02820c700 1 --2- 172.21.15.29:0/1979686630 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb0200fe440 0x7fb0200fe850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:14:38.171 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:14:38.170+0000 7fb02820c700 1 -- 172.21.15.29:0/1979686630 >> 172.21.15.29:0/1979686630 conn(0x7fb0200f6be0 msgr2=0x7fb0200f9010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:14:38.171 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:14:38.170+0000 7fb02820c700 1 -- 172.21.15.29:0/1979686630 shutdown_connections 2024-03-18T01:14:38.171 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:14:38.170+0000 7fb02820c700 1 -- 172.21.15.29:0/1979686630 wait complete. 2024-03-18T01:14:38.172 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:14:38.170+0000 7fb02820c700 1 Processor -- start 2024-03-18T01:14:38.172 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:14:38.170+0000 7fb02820c700 1 -- start start 2024-03-18T01:14:38.173 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:14:38.171+0000 7fb02820c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb0200fe440 0x7fb02010fb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:14:38.173 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:14:38.171+0000 7fb02820c700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb0200fed90 0x7fb020110090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:14:38.173 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:14:38.171+0000 7fb02820c700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb020110690 con 0x7fb0200fe440 2024-03-18T01:14:38.173 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:14:38.171+0000 7fb02820c700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb020110800 con 0x7fb0200fed90 2024-03-18T01:14:38.173 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:14:38.171+0000 7fb025fa8700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb0200fe440 0x7fb02010fb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:14:38.173 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:14:38.171+0000 7fb025fa8700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb0200fe440 0x7fb02010fb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:34012/0 (socket says 172.21.15.29:34012) 2024-03-18T01:14:38.173 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:14:38.171+0000 7fb025fa8700 1 -- 172.21.15.29:0/1228939827 learned_addr learned my addr 172.21.15.29:0/1228939827 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-18T01:14:38.173 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:14:38.171+0000 7fb0257a7700 1 --2- 172.21.15.29:0/1228939827 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb0200fed90 0x7fb020110090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:14:38.173 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:14:38.171+0000 7fb025fa8700 1 -- 172.21.15.29:0/1228939827 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb0200fed90 msgr2=0x7fb020110090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:14:38.173 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:14:38.171+0000 7fb025fa8700 1 --2- 172.21.15.29:0/1228939827 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb0200fed90 0x7fb020110090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:14:38.173 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:14:38.171+0000 7fb025fa8700 1 -- 172.21.15.29:0/1228939827 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb014005040 con 0x7fb0200fe440 2024-03-18T01:14:38.173 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:14:38.171+0000 7fb025fa8700 1 --2- 172.21.15.29:0/1228939827 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb0200fe440 0x7fb02010fb50 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fb010001d80 tx=0x7fb010009ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:14:38.173 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:14:38.171+0000 7fb01e7fc700 1 -- 172.21.15.29:0/1228939827 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb010007d10 con 0x7fb0200fe440 2024-03-18T01:14:38.173 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:14:38.172+0000 7fb01e7fc700 1 -- 172.21.15.29:0/1228939827 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb010001e80 con 0x7fb0200fe440 2024-03-18T01:14:38.174 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:14:38.172+0000 7fb02820c700 1 -- 172.21.15.29:0/1228939827 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb020115260 con 0x7fb0200fe440 2024-03-18T01:14:38.174 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:14:38.172+0000 7fb01e7fc700 1 -- 172.21.15.29:0/1228939827 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb010014640 con 0x7fb0200fe440 2024-03-18T01:14:38.174 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:14:38.172+0000 7fb02820c700 1 -- 172.21.15.29:0/1228939827 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb020115820 con 0x7fb0200fe440 2024-03-18T01:14:38.175 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:14:38.173+0000 7fb02820c700 1 -- 172.21.15.29:0/1228939827 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb020101b80 con 0x7fb0200fe440 2024-03-18T01:14:38.175 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:14:38.173+0000 7fb01e7fc700 1 -- 172.21.15.29:0/1228939827 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 18) v1 ==== 86565+0+0 (secure 0 0 0) 0x7fb01000e960 con 0x7fb0200fe440 2024-03-18T01:14:38.175 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:14:38.174+0000 7fb01e7fc700 1 --2- 172.21.15.29:0/1228939827 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7fb008069210 0x7fb00806b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:14:38.175 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:14:38.174+0000 7fb01e7fc700 1 -- 172.21.15.29:0/1228939827 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1978+0+0 (secure 0 0 0) 0x7fb010085f80 con 0x7fb0200fe440 2024-03-18T01:14:38.175 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:14:38.174+0000 7fb0257a7700 1 --2- 172.21.15.29:0/1228939827 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7fb008069210 0x7fb00806b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:14:38.175 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:14:38.175+0000 7fb0257a7700 1 --2- 172.21.15.29:0/1228939827 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7fb008069210 0x7fb00806b6c0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fb014004890 tx=0x7fb014009e30 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:14:38.179 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:14:38.179+0000 7fb01e7fc700 1 -- 172.21.15.29:0/1228939827 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175520 (secure 0 0 0) 0x7fb01004ddb0 con 0x7fb0200fe440 2024-03-18T01:14:38.387 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:14:38.386+0000 7fb02820c700 1 -- 172.21.15.29:0/1228939827 --> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi029:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7fb02002b310 con 0x7fb008069210 2024-03-18T01:14:39.319 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:14:39 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:14:39.319 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:14:39 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:14:39.319 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:14:39 smithi029 ceph-mon[29331]: from='client.14264 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi029:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T01:14:39.319 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:14:39 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T01:14:39.319 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:14:39 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T01:14:39.319 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:14:39 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:14:39.319 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:14:39 smithi029 ceph-mon[29331]: pgmap v42: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-03-18T01:14:39.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:14:39 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:14:39.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:14:39 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:14:39.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:14:39 smithi090 ceph-mon[36989]: from='client.14264 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi029:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T01:14:39.527 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:14:39 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T01:14:39.527 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:14:39 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T01:14:39.527 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:14:39 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:14:39.527 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:14:39 smithi090 ceph-mon[36989]: pgmap v42: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-03-18T01:14:41.150 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:14:41 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-18T01:14:41.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:14:41 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-18T01:14:42.424 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:14:42 smithi029 ceph-mon[29331]: pgmap v43: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-03-18T01:14:42.424 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:14:42 smithi029 ceph-mon[29331]: from='client.? 172.21.15.29:0/2190829889' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3b1f0295-4dd6-4bb1-9847-93bd98677db6"}]: dispatch 2024-03-18T01:14:42.424 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:14:42 smithi029 ceph-mon[29331]: from='client.? 172.21.15.29:0/2190829889' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3b1f0295-4dd6-4bb1-9847-93bd98677db6"}]': finished 2024-03-18T01:14:42.424 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:14:42 smithi029 ceph-mon[29331]: osdmap e11: 2 total, 1 up, 2 in 2024-03-18T01:14:42.424 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:14:42 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-18T01:14:42.525 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:14:42 smithi090 ceph-mon[36989]: pgmap v43: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-03-18T01:14:42.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:14:42 smithi090 ceph-mon[36989]: from='client.? 172.21.15.29:0/2190829889' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3b1f0295-4dd6-4bb1-9847-93bd98677db6"}]: dispatch 2024-03-18T01:14:42.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:14:42 smithi090 ceph-mon[36989]: from='client.? 172.21.15.29:0/2190829889' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3b1f0295-4dd6-4bb1-9847-93bd98677db6"}]': finished 2024-03-18T01:14:42.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:14:42 smithi090 ceph-mon[36989]: osdmap e11: 2 total, 1 up, 2 in 2024-03-18T01:14:42.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:14:42 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-18T01:14:43.319 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:14:43 smithi029 ceph-mon[29331]: from='client.? 172.21.15.29:0/892192841' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T01:14:43.525 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:14:43 smithi090 ceph-mon[36989]: from='client.? 172.21.15.29:0/892192841' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T01:14:44.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:14:44 smithi090 ceph-mon[36989]: pgmap v45: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-03-18T01:14:44.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:14:44 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:14:44.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:14:44 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:14:44.527 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:14:44 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:14:44.527 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:14:44 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T01:14:44.527 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:14:44 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:14:44.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:14:44 smithi029 ceph-mon[29331]: pgmap v45: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-03-18T01:14:44.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:14:44 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:14:44.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:14:44 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:14:44.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:14:44 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:14:44.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:14:44 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T01:14:44.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:14:44 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:14:46.170 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:14:46 smithi029 ceph-mon[29331]: pgmap v46: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-03-18T01:14:46.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:14:46 smithi090 ceph-mon[36989]: pgmap v46: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-03-18T01:14:48.319 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:14:48 smithi029 ceph-mon[29331]: pgmap v47: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-03-18T01:14:48.525 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:14:48 smithi090 ceph-mon[36989]: pgmap v47: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-03-18T01:14:49.319 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:14:49 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-18T01:14:49.319 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:14:49 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:14:49.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:14:49 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-18T01:14:49.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:14:49 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:14:50.198 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:14:50 smithi029 ceph-mon[29331]: pgmap v48: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-03-18T01:14:50.198 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:14:50 smithi029 ceph-mon[29331]: Deploying daemon osd.1 on smithi029 2024-03-18T01:14:50.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:14:50 smithi090 ceph-mon[36989]: pgmap v48: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-03-18T01:14:50.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:14:50 smithi090 ceph-mon[36989]: Deploying daemon osd.1 on smithi029 2024-03-18T01:14:52.319 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:14:52 smithi029 ceph-mon[29331]: pgmap v49: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-03-18T01:14:52.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:14:52 smithi090 ceph-mon[36989]: pgmap v49: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-03-18T01:14:54.284 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:14:54 smithi029 ceph-mon[29331]: pgmap v50: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-03-18T01:14:54.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:14:54 smithi090 ceph-mon[36989]: pgmap v50: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-03-18T01:14:55.982 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:14:55 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:14:55.982 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:14:55 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:14:55.982 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:14:55 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T01:14:55.982 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:14:55 smithi090 ceph-mon[36989]: pgmap v51: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-03-18T01:14:56.069 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:14:55 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:14:56.069 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:14:55 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:14:56.069 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:14:55 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T01:14:56.069 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:14:55 smithi029 ceph-mon[29331]: pgmap v51: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-03-18T01:14:56.819 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:14:56 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-18T01:14:56.819 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:14:56 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:14:56.819 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:14:56 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:14:57.026 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:14:56 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-18T01:14:57.026 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:14:56 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:14:57.026 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:14:56 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:14:58.026 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:14:57 smithi090 ceph-mon[36989]: pgmap v52: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-03-18T01:14:58.069 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:14:57 smithi029 ceph-mon[29331]: pgmap v52: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-03-18T01:14:58.779 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:14:58 smithi029 ceph-mon[29331]: from='osd.1 [v2:172.21.15.29:6810/2615748531,v1:172.21.15.29:6811/2615748531]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-18T01:14:58.844 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:14:58.843+0000 7fb01e7fc700 1 -- 172.21.15.29:0/1228939827 <== mgr.14208 v2:172.21.15.29:6800/1641454714 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fb02002b310 con 0x7fb008069210 2024-03-18T01:14:58.844 INFO:teuthology.orchestra.run.smithi029.stdout:Created osd(s) 1 on host 'smithi029' 2024-03-18T01:14:58.846 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:14:58.845+0000 7fb02820c700 1 -- 172.21.15.29:0/1228939827 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7fb008069210 msgr2=0x7fb00806b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:14:58.846 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:14:58.845+0000 7fb02820c700 1 --2- 172.21.15.29:0/1228939827 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7fb008069210 0x7fb00806b6c0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fb014004890 tx=0x7fb014009e30 comp rx=0 tx=0).stop 2024-03-18T01:14:58.846 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:14:58.845+0000 7fb02820c700 1 -- 172.21.15.29:0/1228939827 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb0200fe440 msgr2=0x7fb02010fb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:14:58.847 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:14:58.845+0000 7fb02820c700 1 --2- 172.21.15.29:0/1228939827 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb0200fe440 0x7fb02010fb50 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fb010001d80 tx=0x7fb010009ec0 comp rx=0 tx=0).stop 2024-03-18T01:14:58.847 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:14:58.845+0000 7fb02820c700 1 -- 172.21.15.29:0/1228939827 shutdown_connections 2024-03-18T01:14:58.847 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:14:58.845+0000 7fb02820c700 1 --2- 172.21.15.29:0/1228939827 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7fb008069210 0x7fb00806b6c0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:14:58.847 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:14:58.845+0000 7fb02820c700 1 --2- 172.21.15.29:0/1228939827 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb0200fe440 0x7fb02010fb50 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:14:58.847 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:14:58.845+0000 7fb02820c700 1 --2- 172.21.15.29:0/1228939827 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb0200fed90 0x7fb020110090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:14:58.847 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:14:58.845+0000 7fb02820c700 1 -- 172.21.15.29:0/1228939827 >> 172.21.15.29:0/1228939827 conn(0x7fb0200f6be0 msgr2=0x7fb0200f81a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:14:58.847 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:14:58.845+0000 7fb02820c700 1 -- 172.21.15.29:0/1228939827 shutdown_connections 2024-03-18T01:14:58.847 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:14:58.845+0000 7fb02820c700 1 -- 172.21.15.29:0/1228939827 wait complete. 2024-03-18T01:14:59.026 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:14:58 smithi090 ceph-mon[36989]: from='osd.1 [v2:172.21.15.29:6810/2615748531,v1:172.21.15.29:6811/2615748531]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-18T01:14:59.567 DEBUG:teuthology.orchestra.run.smithi029:osd.1> sudo journalctl -f -n 0 -u ceph-37f51182-e4c4-11ee-95c9-87774f69a715@osd.1.service 2024-03-18T01:14:59.570 INFO:tasks.cephadm:Deploying osd.2 on smithi029 with /dev/vg_nvme/lv_2... 2024-03-18T01:14:59.570 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 37f51182-e4c4-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-18T01:14:59.819 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:14:59 smithi029 ceph-mon[29331]: from='osd.1 [v2:172.21.15.29:6810/2615748531,v1:172.21.15.29:6811/2615748531]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-18T01:14:59.819 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:14:59 smithi029 ceph-mon[29331]: osdmap e12: 2 total, 1 up, 2 in 2024-03-18T01:14:59.819 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:14:59 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-18T01:14:59.819 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:14:59 smithi029 ceph-mon[29331]: from='osd.1 [v2:172.21.15.29:6810/2615748531,v1:172.21.15.29:6811/2615748531]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]: dispatch 2024-03-18T01:14:59.819 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:14:59 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:14:59.819 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:14:59 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:14:59.819 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:14:59 smithi029 ceph-mon[29331]: pgmap v54: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-03-18T01:14:59.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:14:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:14:59.632+0000 7fa2c1848700 -1 osd.1 0 waiting for initial osdmap 2024-03-18T01:14:59.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:14:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:14:59.638+0000 7fa2bd645700 -1 osd.1 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-18T01:15:00.026 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:14:59 smithi090 ceph-mon[36989]: from='osd.1 [v2:172.21.15.29:6810/2615748531,v1:172.21.15.29:6811/2615748531]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-18T01:15:00.026 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:14:59 smithi090 ceph-mon[36989]: osdmap e12: 2 total, 1 up, 2 in 2024-03-18T01:15:00.026 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:14:59 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-18T01:15:00.026 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:14:59 smithi090 ceph-mon[36989]: from='osd.1 [v2:172.21.15.29:6810/2615748531,v1:172.21.15.29:6811/2615748531]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]: dispatch 2024-03-18T01:15:00.026 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:14:59 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:15:00.026 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:14:59 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:15:00.026 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:14:59 smithi090 ceph-mon[36989]: pgmap v54: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-03-18T01:15:00.268 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/37f51182-e4c4-11ee-95c9-87774f69a715/mon.smithi029/config 2024-03-18T01:15:00.729 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:15:00 smithi029 ceph-mon[29331]: from='osd.1 [v2:172.21.15.29:6810/2615748531,v1:172.21.15.29:6811/2615748531]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]': finished 2024-03-18T01:15:00.729 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:15:00 smithi029 ceph-mon[29331]: osdmap e13: 2 total, 1 up, 2 in 2024-03-18T01:15:00.729 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:15:00 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-18T01:15:00.729 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:15:00 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-18T01:15:01.026 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:15:00 smithi090 ceph-mon[36989]: from='osd.1 [v2:172.21.15.29:6810/2615748531,v1:172.21.15.29:6811/2615748531]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]': finished 2024-03-18T01:15:01.026 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:15:00 smithi090 ceph-mon[36989]: osdmap e13: 2 total, 1 up, 2 in 2024-03-18T01:15:01.026 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:15:00 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-18T01:15:01.026 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:15:00 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-18T01:15:01.819 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:15:01 smithi029 ceph-mon[29331]: purged_snaps scrub starts 2024-03-18T01:15:01.819 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:15:01 smithi029 ceph-mon[29331]: purged_snaps scrub ok 2024-03-18T01:15:01.819 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:15:01 smithi029 ceph-mon[29331]: osd.1 [v2:172.21.15.29:6810/2615748531,v1:172.21.15.29:6811/2615748531] boot 2024-03-18T01:15:01.819 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:15:01 smithi029 ceph-mon[29331]: osdmap e14: 2 total, 2 up, 2 in 2024-03-18T01:15:01.819 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:15:01 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-18T01:15:01.819 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:15:01 smithi029 ceph-mon[29331]: pgmap v57: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-03-18T01:15:02.026 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:15:01 smithi090 ceph-mon[36989]: purged_snaps scrub starts 2024-03-18T01:15:02.026 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:15:01 smithi090 ceph-mon[36989]: purged_snaps scrub ok 2024-03-18T01:15:02.026 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:15:01 smithi090 ceph-mon[36989]: osd.1 [v2:172.21.15.29:6810/2615748531,v1:172.21.15.29:6811/2615748531] boot 2024-03-18T01:15:02.026 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:15:01 smithi090 ceph-mon[36989]: osdmap e14: 2 total, 2 up, 2 in 2024-03-18T01:15:02.026 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:15:01 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-18T01:15:02.026 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:15:01 smithi090 ceph-mon[36989]: pgmap v57: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-03-18T01:15:03.026 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:15:02 smithi090 ceph-mon[36989]: osdmap e15: 2 total, 2 up, 2 in 2024-03-18T01:15:03.055 INFO:teuthology.orchestra.run.smithi029.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-18T01:15:03.056 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-18T01:15:03.056 INFO:teuthology.orchestra.run.smithi029.stderr: stderr: 10+0 records in 2024-03-18T01:15:03.056 INFO:teuthology.orchestra.run.smithi029.stderr:10+0 records out 2024-03-18T01:15:03.056 INFO:teuthology.orchestra.run.smithi029.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00890718 s, 1.2 GB/s 2024-03-18T01:15:03.056 INFO:teuthology.orchestra.run.smithi029.stderr:--> Zapping successful for: 2024-03-18T01:15:03.056 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:15:02 smithi029 ceph-mon[29331]: osdmap e15: 2 total, 2 up, 2 in 2024-03-18T01:15:03.798 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:15:03 smithi029 ceph-mon[29331]: pgmap v59: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-03-18T01:15:03.823 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 37f51182-e4c4-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi029:vg_nvme/lv_2 2024-03-18T01:15:04.025 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:15:03 smithi090 ceph-mon[36989]: pgmap v59: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-03-18T01:15:04.174 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/37f51182-e4c4-11ee-95c9-87774f69a715/mon.smithi029/config 2024-03-18T01:15:05.317 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:15:05 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:15:05.317 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:15:05 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:15:05.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:15:05 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:15:05.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:15:05 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:15:06.516 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:15:06 smithi029 ceph-mon[29331]: pgmap v60: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-03-18T01:15:06.517 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:15:06.515+0000 7fc974d35700 1 -- 172.21.15.29:0/1458315017 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc9700662e0 msgr2=0x7fc9700ff9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:15:06.518 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:15:06.515+0000 7fc974d35700 1 --2- 172.21.15.29:0/1458315017 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc9700662e0 0x7fc9700ff9c0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fc968009ce0 tx=0x7fc968004d10 comp rx=0 tx=0).stop 2024-03-18T01:15:06.518 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:15:06.515+0000 7fc974d35700 1 -- 172.21.15.29:0/1458315017 shutdown_connections 2024-03-18T01:15:06.518 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:15:06.515+0000 7fc974d35700 1 --2- 172.21.15.29:0/1458315017 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc9700fff00 0x7fc970102320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:15:06.518 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:15:06.515+0000 7fc974d35700 1 --2- 172.21.15.29:0/1458315017 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc9700662e0 0x7fc9700ff9c0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:15:06.518 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:15:06.515+0000 7fc974d35700 1 -- 172.21.15.29:0/1458315017 >> 172.21.15.29:0/1458315017 conn(0x7fc9700f7430 msgr2=0x7fc9700f9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:15:06.518 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:15:06.515+0000 7fc974d35700 1 -- 172.21.15.29:0/1458315017 shutdown_connections 2024-03-18T01:15:06.518 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:15:06.515+0000 7fc974d35700 1 -- 172.21.15.29:0/1458315017 wait complete. 2024-03-18T01:15:06.518 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:15:06.516+0000 7fc974d35700 1 Processor -- start 2024-03-18T01:15:06.518 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:15:06.516+0000 7fc974d35700 1 -- start start 2024-03-18T01:15:06.518 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:15:06.516+0000 7fc974d35700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc9700662e0 0x7fc97010dbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:15:06.518 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:15:06.516+0000 7fc974d35700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc9700fff00 0x7fc97010e130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:15:06.518 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:15:06.516+0000 7fc974d35700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc97010e730 con 0x7fc9700662e0 2024-03-18T01:15:06.518 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:15:06.516+0000 7fc974d35700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc97010ef80 con 0x7fc9700fff00 2024-03-18T01:15:06.518 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:15:06.516+0000 7fc96e59c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc9700662e0 0x7fc97010dbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:15:06.519 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:15:06.516+0000 7fc96e59c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc9700662e0 0x7fc97010dbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54752/0 (socket says 172.21.15.29:54752) 2024-03-18T01:15:06.519 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:15:06.516+0000 7fc96e59c700 1 -- 172.21.15.29:0/1655558955 learned_addr learned my addr 172.21.15.29:0/1655558955 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-18T01:15:06.519 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:15:06.516+0000 7fc96dd9b700 1 --2- 172.21.15.29:0/1655558955 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc9700fff00 0x7fc97010e130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:15:06.519 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:15:06.516+0000 7fc96e59c700 1 -- 172.21.15.29:0/1655558955 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc9700fff00 msgr2=0x7fc97010e130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:15:06.519 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:15:06.516+0000 7fc96e59c700 1 --2- 172.21.15.29:0/1655558955 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc9700fff00 0x7fc97010e130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:15:06.519 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:15:06.516+0000 7fc96e59c700 1 -- 172.21.15.29:0/1655558955 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc968009940 con 0x7fc9700662e0 2024-03-18T01:15:06.519 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:15:06.516+0000 7fc96dd9b700 1 --2- 172.21.15.29:0/1655558955 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc9700fff00 0x7fc97010e130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T01:15:06.519 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:15:06.517+0000 7fc96e59c700 1 --2- 172.21.15.29:0/1655558955 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc9700662e0 0x7fc97010dbf0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fc968009910 tx=0x7fc968004d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:15:06.519 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:15:06.517+0000 7fc966ffd700 1 -- 172.21.15.29:0/1655558955 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc96800e040 con 0x7fc9700662e0 2024-03-18T01:15:06.519 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:15:06.517+0000 7fc974d35700 1 -- 172.21.15.29:0/1655558955 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc97010f200 con 0x7fc9700662e0 2024-03-18T01:15:06.519 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:15:06.517+0000 7fc966ffd700 1 -- 172.21.15.29:0/1655558955 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc968008ed0 con 0x7fc9700662e0 2024-03-18T01:15:06.519 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:15:06.517+0000 7fc966ffd700 1 -- 172.21.15.29:0/1655558955 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc96801c800 con 0x7fc9700662e0 2024-03-18T01:15:06.519 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:15:06.517+0000 7fc974d35700 1 -- 172.21.15.29:0/1655558955 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc97010f740 con 0x7fc9700662e0 2024-03-18T01:15:06.520 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:15:06.518+0000 7fc974d35700 1 -- 172.21.15.29:0/1655558955 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc97005f5f0 con 0x7fc9700662e0 2024-03-18T01:15:06.520 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:15:06.519+0000 7fc966ffd700 1 -- 172.21.15.29:0/1655558955 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 18) v1 ==== 86565+0+0 (secure 0 0 0) 0x7fc968012410 con 0x7fc9700662e0 2024-03-18T01:15:06.520 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:15:06.519+0000 7fc966ffd700 1 --2- 172.21.15.29:0/1655558955 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7fc95c069210 0x7fc95c06b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:15:06.520 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:15:06.519+0000 7fc966ffd700 1 -- 172.21.15.29:0/1655558955 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 2410+0+0 (secure 0 0 0) 0x7fc9680881a0 con 0x7fc9700662e0 2024-03-18T01:15:06.520 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:15:06.519+0000 7fc96dd9b700 1 --2- 172.21.15.29:0/1655558955 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7fc95c069210 0x7fc95c06b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:15:06.520 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:15:06.520+0000 7fc96dd9b700 1 --2- 172.21.15.29:0/1655558955 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7fc95c069210 0x7fc95c06b6c0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fc958007fd0 tx=0x7fc958009040 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:15:06.524 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:15:06.524+0000 7fc966ffd700 1 -- 172.21.15.29:0/1655558955 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175520 (secure 0 0 0) 0x7fc968058bb0 con 0x7fc9700662e0 2024-03-18T01:15:06.525 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:15:06 smithi090 ceph-mon[36989]: pgmap v60: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-03-18T01:15:06.727 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:15:06.727+0000 7fc974d35700 1 -- 172.21.15.29:0/1655558955 --> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi029:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7fc97010fa90 con 0x7fc95c069210 2024-03-18T01:15:07.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:15:07 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T01:15:07.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:15:07 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T01:15:07.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:15:07 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:15:07.568 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:15:07 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T01:15:07.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:15:07 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T01:15:07.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:15:07 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:15:08.482 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:15:08 smithi029 ceph-mon[29331]: from='client.14282 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi029:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T01:15:08.482 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:15:08 smithi029 ceph-mon[29331]: pgmap v61: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-03-18T01:15:08.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:15:08 smithi090 ceph-mon[36989]: from='client.14282 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi029:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T01:15:08.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:15:08 smithi090 ceph-mon[36989]: pgmap v61: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-03-18T01:15:10.069 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:15:09 smithi029 ceph-mon[29331]: Detected new or changed devices on smithi029 2024-03-18T01:15:10.069 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:15:09 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:15:10.069 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:15:09 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:15:10.069 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:15:09 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-18T01:15:10.069 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:15:09 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-18T01:15:10.069 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:15:09 smithi029 ceph-mon[29331]: Adjusting osd_memory_target on smithi029 to 6434M 2024-03-18T01:15:10.070 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:15:09 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:15:10.070 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:15:09 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:15:10.070 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:15:09 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T01:15:10.070 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:15:09 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:15:10.070 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:15:09 smithi029 ceph-mon[29331]: pgmap v62: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-03-18T01:15:10.070 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:15:09 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T01:15:10.081 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:15:09 smithi090 ceph-mon[36989]: Detected new or changed devices on smithi029 2024-03-18T01:15:10.081 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:15:09 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:15:10.081 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:15:09 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:15:10.081 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:15:09 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-18T01:15:10.081 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:15:09 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-18T01:15:10.081 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:15:09 smithi090 ceph-mon[36989]: Adjusting osd_memory_target on smithi029 to 6434M 2024-03-18T01:15:10.081 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:15:09 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:15:10.081 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:15:09 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:15:10.081 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:15:09 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T01:15:10.081 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:15:09 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:15:10.081 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:15:09 smithi090 ceph-mon[36989]: pgmap v62: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-03-18T01:15:10.082 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:15:09 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T01:15:11.175 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:15:11 smithi029 ceph-mon[29331]: from='client.? 172.21.15.29:0/1652869503' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "826d30cd-7b7d-4f93-9567-c6bf6120dfe0"}]: dispatch 2024-03-18T01:15:11.175 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:15:11 smithi029 ceph-mon[29331]: from='client.? 172.21.15.29:0/1652869503' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "826d30cd-7b7d-4f93-9567-c6bf6120dfe0"}]': finished 2024-03-18T01:15:11.175 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:15:11 smithi029 ceph-mon[29331]: osdmap e16: 3 total, 2 up, 3 in 2024-03-18T01:15:11.176 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:15:11 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-18T01:15:11.176 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:15:11 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-18T01:15:11.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:15:11 smithi090 ceph-mon[36989]: from='client.? 172.21.15.29:0/1652869503' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "826d30cd-7b7d-4f93-9567-c6bf6120dfe0"}]: dispatch 2024-03-18T01:15:11.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:15:11 smithi090 ceph-mon[36989]: from='client.? 172.21.15.29:0/1652869503' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "826d30cd-7b7d-4f93-9567-c6bf6120dfe0"}]': finished 2024-03-18T01:15:11.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:15:11 smithi090 ceph-mon[36989]: osdmap e16: 3 total, 2 up, 3 in 2024-03-18T01:15:11.527 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:15:11 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-18T01:15:11.527 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:15:11 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-18T01:15:12.406 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:15:12 smithi029 ceph-mon[29331]: pgmap v64: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-03-18T01:15:12.406 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:15:12 smithi029 ceph-mon[29331]: from='client.? 172.21.15.29:0/3205218002' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T01:15:12.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:15:12 smithi090 ceph-mon[36989]: pgmap v64: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-03-18T01:15:12.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:15:12 smithi090 ceph-mon[36989]: from='client.? 172.21.15.29:0/3205218002' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T01:15:14.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:15:14 smithi090 ceph-mon[36989]: pgmap v65: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-03-18T01:15:14.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:15:14 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:15:14.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:15:14 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:15:14.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:15:14 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:15:14.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:15:14 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T01:15:14.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:15:14 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:15:14.557 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:15:14 smithi029 ceph-mon[29331]: pgmap v65: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-03-18T01:15:14.558 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:15:14 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:15:14.558 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:15:14 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:15:14.558 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:15:14 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:15:14.558 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:15:14 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T01:15:14.558 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:15:14 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:15:16.334 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:15:16 smithi029 ceph-mon[29331]: pgmap v66: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-03-18T01:15:16.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:15:16 smithi090 ceph-mon[36989]: pgmap v66: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-03-18T01:15:18.319 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:15:18 smithi029 ceph-mon[29331]: pgmap v67: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-03-18T01:15:18.319 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:15:18 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-18T01:15:18.319 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:15:18 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:15:18.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:15:18 smithi090 ceph-mon[36989]: pgmap v67: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-03-18T01:15:18.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:15:18 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-18T01:15:18.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:15:18 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:15:19.319 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:15:19 smithi029 ceph-mon[29331]: Deploying daemon osd.2 on smithi029 2024-03-18T01:15:19.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:15:19 smithi090 ceph-mon[36989]: Deploying daemon osd.2 on smithi029 2024-03-18T01:15:20.319 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:15:20 smithi029 ceph-mon[29331]: pgmap v68: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-03-18T01:15:20.525 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:15:20 smithi090 ceph-mon[36989]: pgmap v68: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-03-18T01:15:22.319 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:15:22 smithi029 ceph-mon[29331]: pgmap v69: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-03-18T01:15:22.525 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:15:22 smithi090 ceph-mon[36989]: pgmap v69: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-03-18T01:15:24.123 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:15:24 smithi029 ceph-mon[29331]: pgmap v70: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-03-18T01:15:24.123 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:15:24 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:15:24.123 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:15:24 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:15:24.123 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:15:24 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T01:15:24.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:15:24 smithi090 ceph-mon[36989]: pgmap v70: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-03-18T01:15:24.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:15:24 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:15:24.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:15:24 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:15:24.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:15:24 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T01:15:26.319 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:15:26 smithi029 ceph-mon[29331]: pgmap v71: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-03-18T01:15:26.319 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:15:26 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-18T01:15:26.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:15:26 smithi090 ceph-mon[36989]: pgmap v71: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-03-18T01:15:26.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:15:26 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-18T01:15:26.659 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:15:26.659+0000 7fc966ffd700 1 -- 172.21.15.29:0/1655558955 <== mgr.14208 v2:172.21.15.29:6800/1641454714 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fc97010fa90 con 0x7fc95c069210 2024-03-18T01:15:26.660 INFO:teuthology.orchestra.run.smithi029.stdout:Created osd(s) 2 on host 'smithi029' 2024-03-18T01:15:26.662 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:15:26.661+0000 7fc974d35700 1 -- 172.21.15.29:0/1655558955 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7fc95c069210 msgr2=0x7fc95c06b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:15:26.662 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:15:26.662+0000 7fc974d35700 1 --2- 172.21.15.29:0/1655558955 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7fc95c069210 0x7fc95c06b6c0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fc958007fd0 tx=0x7fc958009040 comp rx=0 tx=0).stop 2024-03-18T01:15:26.662 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:15:26.662+0000 7fc974d35700 1 -- 172.21.15.29:0/1655558955 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc9700662e0 msgr2=0x7fc97010dbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:15:26.662 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:15:26.662+0000 7fc974d35700 1 --2- 172.21.15.29:0/1655558955 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc9700662e0 0x7fc97010dbf0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fc968009910 tx=0x7fc968004d10 comp rx=0 tx=0).stop 2024-03-18T01:15:26.662 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:15:26.662+0000 7fc974d35700 1 -- 172.21.15.29:0/1655558955 shutdown_connections 2024-03-18T01:15:26.663 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:15:26.662+0000 7fc974d35700 1 --2- 172.21.15.29:0/1655558955 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7fc95c069210 0x7fc95c06b6c0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:15:26.663 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:15:26.662+0000 7fc974d35700 1 --2- 172.21.15.29:0/1655558955 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc9700662e0 0x7fc97010dbf0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:15:26.663 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:15:26.662+0000 7fc974d35700 1 --2- 172.21.15.29:0/1655558955 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc9700fff00 0x7fc97010e130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:15:26.663 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:15:26.662+0000 7fc974d35700 1 -- 172.21.15.29:0/1655558955 >> 172.21.15.29:0/1655558955 conn(0x7fc9700f7430 msgr2=0x7fc9700f7850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:15:26.663 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:15:26.662+0000 7fc974d35700 1 -- 172.21.15.29:0/1655558955 shutdown_connections 2024-03-18T01:15:26.663 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:15:26.662+0000 7fc974d35700 1 -- 172.21.15.29:0/1655558955 wait complete. 2024-03-18T01:15:27.323 DEBUG:teuthology.orchestra.run.smithi029:osd.2> sudo journalctl -f -n 0 -u ceph-37f51182-e4c4-11ee-95c9-87774f69a715@osd.2.service 2024-03-18T01:15:27.326 INFO:tasks.cephadm:Deploying osd.3 on smithi090 with /dev/vg_nvme/lv_4... 2024-03-18T01:15:27.326 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 37f51182-e4c4-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-18T01:15:27.569 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:15:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:15:27.362+0000 7f0266f06540 -1 osd.2 0 log_to_monitors true 2024-03-18T01:15:27.588 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/37f51182-e4c4-11ee-95c9-87774f69a715/mon.smithi090/config 2024-03-18T01:15:27.909 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:15:27 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:15:27.910 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:15:27 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:15:27.910 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:15:27 smithi090 ceph-mon[36989]: pgmap v72: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-03-18T01:15:27.910 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:15:27 smithi090 ceph-mon[36989]: from='osd.2 [v2:172.21.15.29:6818/576060095,v1:172.21.15.29:6819/576060095]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-18T01:15:28.069 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:15:27 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:15:28.069 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:15:27 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:15:28.069 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:15:27 smithi029 ceph-mon[29331]: pgmap v72: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-03-18T01:15:28.069 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:15:27 smithi029 ceph-mon[29331]: from='osd.2 [v2:172.21.15.29:6818/576060095,v1:172.21.15.29:6819/576060095]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-18T01:15:28.794 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:15:28 smithi090 ceph-mon[36989]: from='osd.2 [v2:172.21.15.29:6818/576060095,v1:172.21.15.29:6819/576060095]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-18T01:15:28.794 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:15:28 smithi090 ceph-mon[36989]: osdmap e17: 3 total, 2 up, 3 in 2024-03-18T01:15:28.794 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:15:28 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-18T01:15:28.795 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:15:28 smithi090 ceph-mon[36989]: from='osd.2 [v2:172.21.15.29:6818/576060095,v1:172.21.15.29:6819/576060095]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]: dispatch 2024-03-18T01:15:29.006 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:15:28 smithi029 ceph-mon[29331]: from='osd.2 [v2:172.21.15.29:6818/576060095,v1:172.21.15.29:6819/576060095]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-18T01:15:29.006 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:15:28 smithi029 ceph-mon[29331]: osdmap e17: 3 total, 2 up, 3 in 2024-03-18T01:15:29.006 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:15:28 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-18T01:15:29.006 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:15:28 smithi029 ceph-mon[29331]: from='osd.2 [v2:172.21.15.29:6818/576060095,v1:172.21.15.29:6819/576060095]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]: dispatch 2024-03-18T01:15:29.007 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:15:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:15:28.643+0000 7f025d6d8700 -1 osd.2 0 waiting for initial osdmap 2024-03-18T01:15:29.007 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:15:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:15:28.648+0000 7f0258cd4700 -1 osd.2 18 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-18T01:15:29.330 INFO:teuthology.orchestra.run.smithi090.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-18T01:15:29.330 INFO:teuthology.orchestra.run.smithi090.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-18T01:15:29.330 INFO:teuthology.orchestra.run.smithi090.stderr: stderr: 10+0 records in 2024-03-18T01:15:29.330 INFO:teuthology.orchestra.run.smithi090.stderr:10+0 records out 2024-03-18T01:15:29.330 INFO:teuthology.orchestra.run.smithi090.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0131779 s, 796 MB/s 2024-03-18T01:15:29.330 INFO:teuthology.orchestra.run.smithi090.stderr:--> Zapping successful for: 2024-03-18T01:15:29.907 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:15:29 smithi029 ceph-mon[29331]: from='osd.2 [v2:172.21.15.29:6818/576060095,v1:172.21.15.29:6819/576060095]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]': finished 2024-03-18T01:15:29.907 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:15:29 smithi029 ceph-mon[29331]: osdmap e18: 3 total, 2 up, 3 in 2024-03-18T01:15:29.907 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:15:29 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-18T01:15:29.907 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:15:29 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-18T01:15:29.907 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:15:29 smithi029 ceph-mon[29331]: pgmap v75: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-03-18T01:15:29.932 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:15:29 smithi090 ceph-mon[36989]: from='osd.2 [v2:172.21.15.29:6818/576060095,v1:172.21.15.29:6819/576060095]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]': finished 2024-03-18T01:15:29.932 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:15:29 smithi090 ceph-mon[36989]: osdmap e18: 3 total, 2 up, 3 in 2024-03-18T01:15:29.932 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:15:29 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-18T01:15:29.932 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:15:29 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-18T01:15:29.932 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:15:29 smithi090 ceph-mon[36989]: pgmap v75: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-03-18T01:15:29.958 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 37f51182-e4c4-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi090:vg_nvme/lv_4 2024-03-18T01:15:30.210 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/37f51182-e4c4-11ee-95c9-87774f69a715/mon.smithi090/config 2024-03-18T01:15:30.648 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:15:30 smithi029 ceph-mon[29331]: purged_snaps scrub starts 2024-03-18T01:15:30.648 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:15:30 smithi029 ceph-mon[29331]: purged_snaps scrub ok 2024-03-18T01:15:30.648 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:15:30 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-18T01:15:30.649 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:15:30 smithi029 ceph-mon[29331]: osd.2 [v2:172.21.15.29:6818/576060095,v1:172.21.15.29:6819/576060095] boot 2024-03-18T01:15:30.649 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:15:30 smithi029 ceph-mon[29331]: osdmap e19: 3 total, 3 up, 3 in 2024-03-18T01:15:30.649 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:15:30 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-18T01:15:30.649 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:15:30 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:15:30.649 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:15:30 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:15:31.026 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:15:30 smithi090 ceph-mon[36989]: purged_snaps scrub starts 2024-03-18T01:15:31.026 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:15:30 smithi090 ceph-mon[36989]: purged_snaps scrub ok 2024-03-18T01:15:31.026 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:15:30 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-18T01:15:31.026 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:15:30 smithi090 ceph-mon[36989]: osd.2 [v2:172.21.15.29:6818/576060095,v1:172.21.15.29:6819/576060095] boot 2024-03-18T01:15:31.026 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:15:30 smithi090 ceph-mon[36989]: osdmap e19: 3 total, 3 up, 3 in 2024-03-18T01:15:31.026 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:15:30 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-18T01:15:31.026 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:15:30 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:15:31.026 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:15:30 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:15:31.696 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:15:31.694+0000 7f510d034700 1 -- 172.21.15.90:0/2338025868 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f51080fe430 msgr2=0x7f51080fe840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:15:31.696 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:15:31.694+0000 7f510d034700 1 --2- 172.21.15.90:0/2338025868 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f51080fe430 0x7f51080fe840 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f5100009ce0 tx=0x7f5100004d10 comp rx=0 tx=0).stop 2024-03-18T01:15:31.696 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:15:31.694+0000 7f510d034700 1 -- 172.21.15.90:0/2338025868 shutdown_connections 2024-03-18T01:15:31.696 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:15:31.694+0000 7f510d034700 1 --2- 172.21.15.90:0/2338025868 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f51080fed80 0x7f51080faec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:15:31.697 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:15:31.694+0000 7f510d034700 1 --2- 172.21.15.90:0/2338025868 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f51080fe430 0x7f51080fe840 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:15:31.697 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:15:31.694+0000 7f510d034700 1 -- 172.21.15.90:0/2338025868 >> 172.21.15.90:0/2338025868 conn(0x7f51080f6bd0 msgr2=0x7f51080f9000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:15:31.697 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:15:31.694+0000 7f510d034700 1 -- 172.21.15.90:0/2338025868 shutdown_connections 2024-03-18T01:15:31.697 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:15:31.694+0000 7f510d034700 1 -- 172.21.15.90:0/2338025868 wait complete. 2024-03-18T01:15:31.697 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:15:31.695+0000 7f510d034700 1 Processor -- start 2024-03-18T01:15:31.697 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:15:31.695+0000 7f510d034700 1 -- start start 2024-03-18T01:15:31.697 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:15:31.695+0000 7f510d034700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f51080fe430 0x7f51080fda40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:15:31.697 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:15:31.695+0000 7f510d034700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f51080fed80 0x7f51080fc0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:15:31.697 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:15:31.695+0000 7f510d034700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51080fe040 con 0x7f51080fed80 2024-03-18T01:15:31.697 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:15:31.695+0000 7f510d034700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51080fc630 con 0x7f51080fe430 2024-03-18T01:15:31.697 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:15:31.695+0000 7f5106d9d700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f51080fe430 0x7f51080fda40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:15:31.697 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:15:31.695+0000 7f5106d9d700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f51080fe430 0x7f51080fda40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:59434/0 (socket says 172.21.15.90:59434) 2024-03-18T01:15:31.698 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:15:31.695+0000 7f5106d9d700 1 -- 172.21.15.90:0/2053815565 learned_addr learned my addr 172.21.15.90:0/2053815565 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-18T01:15:31.698 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:15:31.695+0000 7f510659c700 1 --2- 172.21.15.90:0/2053815565 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f51080fed80 0x7f51080fc0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:15:31.698 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:15:31.695+0000 7f5106d9d700 1 -- 172.21.15.90:0/2053815565 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f51080fed80 msgr2=0x7f51080fc0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:15:31.698 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:15:31.695+0000 7f5106d9d700 1 --2- 172.21.15.90:0/2053815565 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f51080fed80 0x7f51080fc0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:15:31.698 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:15:31.695+0000 7f5106d9d700 1 -- 172.21.15.90:0/2053815565 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5100009940 con 0x7f51080fe430 2024-03-18T01:15:31.698 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:15:31.695+0000 7f5106d9d700 1 --2- 172.21.15.90:0/2053815565 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f51080fe430 0x7f51080fda40 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f510000bfd0 tx=0x7f5100004d10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:15:31.698 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:15:31.695+0000 7f50ef7fe700 1 -- 172.21.15.90:0/2053815565 <== mon.1 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f510000e040 con 0x7f51080fe430 2024-03-18T01:15:31.698 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:15:31.695+0000 7f510d034700 1 -- 172.21.15.90:0/2053815565 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51080fc880 con 0x7f51080fe430 2024-03-18T01:15:31.698 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:15:31.696+0000 7f50ef7fe700 1 -- 172.21.15.90:0/2053815565 <== mon.1 v2:172.21.15.90:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f510001cd00 con 0x7f51080fe430 2024-03-18T01:15:31.698 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:15:31.696+0000 7f50ef7fe700 1 -- 172.21.15.90:0/2053815565 <== mon.1 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5100012c40 con 0x7f51080fe430 2024-03-18T01:15:31.698 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:15:31.696+0000 7f510d034700 1 -- 172.21.15.90:0/2053815565 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51080fce40 con 0x7f51080fe430 2024-03-18T01:15:31.699 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:15:31.697+0000 7f510d034700 1 -- 172.21.15.90:0/2053815565 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f510804c710 con 0x7f51080fe430 2024-03-18T01:15:31.699 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:15:31.697+0000 7f50ef7fe700 1 -- 172.21.15.90:0/2053815565 <== mon.1 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 18) v1 ==== 86565+0+0 (secure 0 0 0) 0x7f51000124a0 con 0x7f51080fe430 2024-03-18T01:15:31.699 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:15:31.697+0000 7f50ef7fe700 1 --2- 172.21.15.90:0/2053815565 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f50f0069210 0x7f50f006b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:15:31.699 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:15:31.697+0000 7f50ef7fe700 1 -- 172.21.15.90:0/2053815565 <== mon.1 v2:172.21.15.90:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 3217+0+0 (secure 0 0 0) 0x7f5100088b00 con 0x7f51080fe430 2024-03-18T01:15:31.700 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:15:31.697+0000 7f510659c700 1 --2- 172.21.15.90:0/2053815565 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f50f0069210 0x7f50f006b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:15:31.700 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:15:31.698+0000 7f510659c700 1 --2- 172.21.15.90:0/2053815565 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f50f0069210 0x7f50f006b6c0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f50fc009b80 tx=0x7f50fc00b040 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:15:31.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:15:31.702+0000 7f50ef7fe700 1 -- 172.21.15.90:0/2053815565 <== mon.1 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175520 (secure 0 0 0) 0x7f51000591e0 con 0x7f51080fe430 2024-03-18T01:15:31.775 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:15:31 smithi090 ceph-mon[36989]: pgmap v77: 0 pgs: ; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-03-18T01:15:31.776 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:15:31 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-03-18T01:15:31.776 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:15:31 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-03-18T01:15:31.776 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:15:31 smithi090 ceph-mon[36989]: osdmap e20: 3 total, 3 up, 3 in 2024-03-18T01:15:31.776 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:15:31 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-18T01:15:31.896 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:15:31.895+0000 7f510d034700 1 -- 172.21.15.90:0/2053815565 --> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi090:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7f5108102480 con 0x7f50f0069210 2024-03-18T01:15:31.971 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:15:31 smithi029 ceph-mon[29331]: pgmap v77: 0 pgs: ; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-03-18T01:15:31.971 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:15:31 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-03-18T01:15:31.971 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:15:31 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-03-18T01:15:31.971 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:15:31 smithi029 ceph-mon[29331]: osdmap e20: 3 total, 3 up, 3 in 2024-03-18T01:15:31.971 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:15:31 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-18T01:15:32.904 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:15:32 smithi029 ceph-mon[29331]: from='client.24133 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi090:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T01:15:32.904 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:15:32 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T01:15:32.904 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:15:32 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T01:15:32.904 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:15:32 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:15:32.904 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:15:32 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-18T01:15:32.904 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:15:32 smithi029 ceph-mon[29331]: osdmap e21: 3 total, 3 up, 3 in 2024-03-18T01:15:33.026 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:15:32 smithi090 ceph-mon[36989]: from='client.24133 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi090:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T01:15:33.026 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:15:32 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T01:15:33.026 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:15:32 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T01:15:33.026 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:15:32 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:15:33.026 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:15:32 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-18T01:15:33.026 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:15:32 smithi090 ceph-mon[36989]: osdmap e21: 3 total, 3 up, 3 in 2024-03-18T01:15:33.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:15:33 smithi029 sudo[66521]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-18T01:15:33.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:15:33 smithi029 sudo[66521]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-18T01:15:33.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:15:33 smithi029 sudo[66521]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-18T01:15:33.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:15:33 smithi029 sudo[66521]: pam_unix(sudo:session): session closed for user root 2024-03-18T01:15:33.628 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:15:33 smithi029 sudo[66527]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-18T01:15:33.628 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:15:33 smithi029 sudo[66527]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-18T01:15:33.628 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:15:33 smithi029 sudo[66527]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-18T01:15:33.628 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:15:33 smithi029 sudo[66527]: pam_unix(sudo:session): session closed for user root 2024-03-18T01:15:33.944 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:15:33 smithi029 ceph-mon[29331]: pgmap v80: 1 pgs: 1 creating+peering; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-03-18T01:15:33.944 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:15:33 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:15:33.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:15:33 smithi029 sudo[66531]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-18T01:15:33.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:15:33 smithi029 sudo[66531]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-18T01:15:33.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:15:33 smithi029 sudo[66531]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-18T01:15:33.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:15:33 smithi029 sudo[66531]: pam_unix(sudo:session): session closed for user root 2024-03-18T01:15:34.026 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:15:33 smithi090 ceph-mon[36989]: pgmap v80: 1 pgs: 1 creating+peering; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-03-18T01:15:34.026 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:15:33 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:15:34.213 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:15:33 smithi029 sudo[66632]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-18T01:15:34.213 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:15:33 smithi029 sudo[66632]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-18T01:15:34.213 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:15:33 smithi029 sudo[66632]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-18T01:15:34.213 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:15:33 smithi029 sudo[66632]: pam_unix(sudo:session): session closed for user root 2024-03-18T01:15:34.481 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:15:34 smithi029 sudo[66724]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-18T01:15:34.481 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:15:34 smithi029 sudo[66724]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-18T01:15:34.481 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:15:34 smithi029 sudo[66724]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-18T01:15:34.482 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:15:34 smithi029 sudo[66724]: pam_unix(sudo:session): session closed for user root 2024-03-18T01:15:34.738 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:15:34 smithi029 sudo[66732]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-18T01:15:34.738 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:15:34 smithi029 sudo[66732]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-18T01:15:34.738 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:15:34 smithi029 sudo[66732]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-18T01:15:34.738 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:15:34 smithi029 sudo[66732]: pam_unix(sudo:session): session closed for user root 2024-03-18T01:15:35.011 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:15:34 smithi029 sudo[66750]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-18T01:15:35.011 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:15:34 smithi029 sudo[66750]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-18T01:15:35.011 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:15:34 smithi029 sudo[66750]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-18T01:15:35.011 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:15:34 smithi029 sudo[66750]: pam_unix(sudo:session): session closed for user root 2024-03-18T01:15:35.319 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:15:35 smithi029 sudo[66833]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-18T01:15:35.319 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:15:35 smithi029 sudo[66833]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-18T01:15:35.320 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:15:35 smithi029 sudo[66833]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-18T01:15:35.320 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:15:35 smithi029 sudo[66833]: pam_unix(sudo:session): session closed for user root 2024-03-18T01:15:35.320 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:15:35 smithi029 ceph-mon[29331]: osdmap e22: 3 total, 3 up, 3 in 2024-03-18T01:15:35.320 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:15:35 smithi029 ceph-mon[29331]: Detected new or changed devices on smithi029 2024-03-18T01:15:35.320 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:15:35 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:15:35.320 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:15:35 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:15:35.320 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:15:35 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-18T01:15:35.320 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:15:35 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-18T01:15:35.320 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:15:35 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-18T01:15:35.320 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:15:35 smithi029 ceph-mon[29331]: Adjusting osd_memory_target on smithi029 to 4289M 2024-03-18T01:15:35.320 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:15:35 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:15:35.320 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:15:35 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:15:35.320 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:15:35 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T01:15:35.320 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:15:35 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:15:35.320 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:15:35 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T01:15:35.321 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:15:35 smithi029 ceph-mon[29331]: from='client.? 172.21.15.90:0/1856189449' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cba371da-b198-4b14-a3ac-fcf90d1a91e2"}]: dispatch 2024-03-18T01:15:35.321 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:15:35 smithi029 ceph-mon[29331]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cba371da-b198-4b14-a3ac-fcf90d1a91e2"}]: dispatch 2024-03-18T01:15:35.321 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:15:35 smithi029 ceph-mon[29331]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "cba371da-b198-4b14-a3ac-fcf90d1a91e2"}]': finished 2024-03-18T01:15:35.321 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:15:35 smithi029 ceph-mon[29331]: osdmap e23: 4 total, 3 up, 4 in 2024-03-18T01:15:35.321 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:15:35 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-18T01:15:35.321 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:15:35 smithi029 ceph-mon[29331]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-18T01:15:35.509 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:15:35 smithi090 sudo[40526]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-18T01:15:35.509 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:15:35 smithi090 sudo[40526]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-18T01:15:35.509 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:15:35 smithi090 sudo[40526]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-18T01:15:35.509 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:15:35 smithi090 sudo[40526]: pam_unix(sudo:session): session closed for user root 2024-03-18T01:15:35.524 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:15:35.523+0000 7f50ef7fe700 1 -- 172.21.15.90:0/2053815565 <== mon.1 v2:172.21.15.90:3300/0 7 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f5100055b20 con 0x7f51080fe430 2024-03-18T01:15:35.776 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:15:35 smithi090 sudo[40551]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-18T01:15:35.776 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:15:35 smithi090 sudo[40551]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-18T01:15:35.776 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:15:35 smithi090 sudo[40551]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-18T01:15:35.776 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:15:35 smithi090 sudo[40551]: pam_unix(sudo:session): session closed for user root 2024-03-18T01:15:35.776 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:15:35 smithi090 ceph-mon[36989]: osdmap e22: 3 total, 3 up, 3 in 2024-03-18T01:15:35.776 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:15:35 smithi090 ceph-mon[36989]: Detected new or changed devices on smithi029 2024-03-18T01:15:35.776 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:15:35 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:15:35.777 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:15:35 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:15:35.777 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:15:35 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-18T01:15:35.777 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:15:35 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-18T01:15:35.777 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:15:35 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-18T01:15:35.777 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:15:35 smithi090 ceph-mon[36989]: Adjusting osd_memory_target on smithi029 to 4289M 2024-03-18T01:15:35.777 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:15:35 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:15:35.777 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:15:35 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:15:35.777 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:15:35 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T01:15:35.777 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:15:35 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:15:35.777 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:15:35 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T01:15:35.777 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:15:35 smithi090 ceph-mon[36989]: from='client.? 172.21.15.90:0/1856189449' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cba371da-b198-4b14-a3ac-fcf90d1a91e2"}]: dispatch 2024-03-18T01:15:35.777 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:15:35 smithi090 ceph-mon[36989]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cba371da-b198-4b14-a3ac-fcf90d1a91e2"}]: dispatch 2024-03-18T01:15:35.777 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:15:35 smithi090 ceph-mon[36989]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "cba371da-b198-4b14-a3ac-fcf90d1a91e2"}]': finished 2024-03-18T01:15:35.777 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:15:35 smithi090 ceph-mon[36989]: osdmap e23: 4 total, 3 up, 4 in 2024-03-18T01:15:35.778 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:15:35 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-18T01:15:35.778 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:15:35 smithi090 ceph-mon[36989]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-18T01:15:36.276 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:15:36 smithi090 ceph-mon[36989]: from='client.? 172.21.15.90:0/3016186221' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T01:15:36.276 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:15:36 smithi090 ceph-mon[36989]: pgmap v83: 1 pgs: 1 creating+peering; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-03-18T01:15:36.276 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:15:36 smithi090 ceph-mon[36989]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-18T01:15:36.276 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:15:36 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "mon metadata", "id": "smithi029"}]: dispatch 2024-03-18T01:15:36.276 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:15:36 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "mon metadata", "id": "smithi090"}]: dispatch 2024-03-18T01:15:36.276 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:15:36 smithi090 ceph-mon[36989]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-18T01:15:36.276 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:15:36 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "mon metadata", "id": "smithi029"}]: dispatch 2024-03-18T01:15:36.276 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:15:36 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "mon metadata", "id": "smithi090"}]: dispatch 2024-03-18T01:15:36.276 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:15:36 smithi090 ceph-mon[36989]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-18T01:15:36.277 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:15:36 smithi090 ceph-mon[36989]: mgrmap e19: smithi029.qviwhl(active, since 2m), standbys: smithi090.pnqedy 2024-03-18T01:15:36.277 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:15:36 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:15:36.319 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:15:36 smithi029 ceph-mon[29331]: from='client.? 172.21.15.90:0/3016186221' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T01:15:36.319 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:15:36 smithi029 ceph-mon[29331]: pgmap v83: 1 pgs: 1 creating+peering; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-03-18T01:15:36.319 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:15:36 smithi029 ceph-mon[29331]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-18T01:15:36.319 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:15:36 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "mon metadata", "id": "smithi029"}]: dispatch 2024-03-18T01:15:36.319 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:15:36 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "mon metadata", "id": "smithi090"}]: dispatch 2024-03-18T01:15:36.319 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:15:36 smithi029 ceph-mon[29331]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-18T01:15:36.319 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:15:36 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "mon metadata", "id": "smithi029"}]: dispatch 2024-03-18T01:15:36.319 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:15:36 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "mon metadata", "id": "smithi090"}]: dispatch 2024-03-18T01:15:36.319 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:15:36 smithi029 ceph-mon[29331]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-18T01:15:36.319 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:15:36 smithi029 ceph-mon[29331]: mgrmap e19: smithi029.qviwhl(active, since 2m), standbys: smithi090.pnqedy 2024-03-18T01:15:36.320 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:15:36 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:15:38.276 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:15:38 smithi090 ceph-mon[36989]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 873 MiB used, 267 GiB / 268 GiB avail 2024-03-18T01:15:38.276 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:15:38 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:15:38.276 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:15:38 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:15:38.319 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:15:38 smithi029 ceph-mon[29331]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 873 MiB used, 267 GiB / 268 GiB avail 2024-03-18T01:15:38.319 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:15:38 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:15:38.319 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:15:38 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:15:39.276 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:15:39 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:15:39.276 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:15:39 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T01:15:39.276 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:15:39 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:15:39.319 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:15:39 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:15:39.319 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:15:39 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T01:15:39.319 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:15:39 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:15:40.178 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:15:40 smithi090 ceph-mon[36989]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-03-18T01:15:40.319 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:15:40 smithi029 ceph-mon[29331]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-03-18T01:15:41.429 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:15:41 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-18T01:15:41.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:15:41 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-18T01:15:42.216 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:15:42 smithi090 ceph-mon[36989]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-03-18T01:15:42.216 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:15:42 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-18T01:15:42.216 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:15:42 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:15:42.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:15:42 smithi029 ceph-mon[29331]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-03-18T01:15:42.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:15:42 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-18T01:15:42.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:15:42 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:15:43.385 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:15:43 smithi090 ceph-mon[36989]: Deploying daemon osd.3 on smithi090 2024-03-18T01:15:43.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:15:43 smithi029 ceph-mon[29331]: Deploying daemon osd.3 on smithi090 2024-03-18T01:15:44.153 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:15:44 smithi090 ceph-mon[36989]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-03-18T01:15:44.470 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:15:44 smithi029 ceph-mon[29331]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-03-18T01:15:46.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:15:46 smithi090 ceph-mon[36989]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-03-18T01:15:46.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:15:46 smithi029 ceph-mon[29331]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-03-18T01:15:47.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:15:47 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:15:47.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:15:47 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:15:47.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:15:47 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T01:15:47.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:15:47 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:15:47.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:15:47 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:15:47.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:15:47 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T01:15:48.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:15:48 smithi090 ceph-mon[36989]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-03-18T01:15:48.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:15:48 smithi029 ceph-mon[29331]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-03-18T01:15:49.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:15:49.456+0000 7f50ef7fe700 1 -- 172.21.15.90:0/2053815565 <== mgr.14208 v2:172.21.15.29:6800/1641454714 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f5108102480 con 0x7f50f0069210 2024-03-18T01:15:49.457 INFO:teuthology.orchestra.run.smithi090.stdout:Created osd(s) 3 on host 'smithi090' 2024-03-18T01:15:49.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:15:49.458+0000 7f510d034700 1 -- 172.21.15.90:0/2053815565 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f50f0069210 msgr2=0x7f50f006b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:15:49.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:15:49.458+0000 7f510d034700 1 --2- 172.21.15.90:0/2053815565 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f50f0069210 0x7f50f006b6c0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f50fc009b80 tx=0x7f50fc00b040 comp rx=0 tx=0).stop 2024-03-18T01:15:49.459 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:15:49.458+0000 7f510d034700 1 -- 172.21.15.90:0/2053815565 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f51080fe430 msgr2=0x7f51080fda40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:15:49.459 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:15:49.458+0000 7f510d034700 1 --2- 172.21.15.90:0/2053815565 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f51080fe430 0x7f51080fda40 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f510000bfd0 tx=0x7f5100004d10 comp rx=0 tx=0).stop 2024-03-18T01:15:49.459 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:15:49.458+0000 7f510d034700 1 -- 172.21.15.90:0/2053815565 shutdown_connections 2024-03-18T01:15:49.459 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:15:49.458+0000 7f510d034700 1 --2- 172.21.15.90:0/2053815565 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f50f0069210 0x7f50f006b6c0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:15:49.459 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:15:49.458+0000 7f510d034700 1 --2- 172.21.15.90:0/2053815565 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f51080fe430 0x7f51080fda40 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:15:49.459 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:15:49.458+0000 7f510d034700 1 --2- 172.21.15.90:0/2053815565 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f51080fed80 0x7f51080fc0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:15:49.459 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:15:49.458+0000 7f510d034700 1 -- 172.21.15.90:0/2053815565 >> 172.21.15.90:0/2053815565 conn(0x7f51080f6bd0 msgr2=0x7f51080f8250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:15:49.459 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:15:49.458+0000 7f510d034700 1 -- 172.21.15.90:0/2053815565 shutdown_connections 2024-03-18T01:15:49.459 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:15:49.458+0000 7f510d034700 1 -- 172.21.15.90:0/2053815565 wait complete. 2024-03-18T01:15:50.361 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:15:50 smithi090 ceph-mon[36989]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-03-18T01:15:50.361 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:15:50 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:15:50.361 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:15:50 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:15:50.361 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:15:50 smithi090 ceph-mon[36989]: from='osd.3 [v2:172.21.15.90:6800/1797049062,v1:172.21.15.90:6801/1797049062]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-18T01:15:50.361 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:15:50 smithi090 ceph-mon[36989]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-18T01:15:50.384 DEBUG:teuthology.orchestra.run.smithi090:osd.3> sudo journalctl -f -n 0 -u ceph-37f51182-e4c4-11ee-95c9-87774f69a715@osd.3.service 2024-03-18T01:15:50.387 INFO:tasks.cephadm:Deploying osd.4 on smithi090 with /dev/vg_nvme/lv_3... 2024-03-18T01:15:50.387 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 37f51182-e4c4-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-18T01:15:50.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:15:50 smithi029 ceph-mon[29331]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-03-18T01:15:50.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:15:50 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:15:50.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:15:50 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:15:50.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:15:50 smithi029 ceph-mon[29331]: from='osd.3 [v2:172.21.15.90:6800/1797049062,v1:172.21.15.90:6801/1797049062]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-18T01:15:50.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:15:50 smithi029 ceph-mon[29331]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-18T01:15:50.687 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/37f51182-e4c4-11ee-95c9-87774f69a715/mon.smithi090/config 2024-03-18T01:15:51.850 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:15:51 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:15:51.850 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:15:51 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:15:51.850 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:15:51 smithi090 ceph-mon[36989]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-03-18T01:15:51.850 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:15:51 smithi090 ceph-mon[36989]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-18T01:15:51.850 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:15:51 smithi090 ceph-mon[36989]: osdmap e24: 4 total, 3 up, 4 in 2024-03-18T01:15:51.850 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:15:51 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-18T01:15:51.850 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:15:51 smithi090 ceph-mon[36989]: from='osd.3 [v2:172.21.15.90:6800/1797049062,v1:172.21.15.90:6801/1797049062]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi090", "root=default"]}]: dispatch 2024-03-18T01:15:51.850 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:15:51 smithi090 ceph-mon[36989]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi090", "root=default"]}]: dispatch 2024-03-18T01:15:52.069 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:15:51 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:15:52.069 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:15:51 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:15:52.069 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:15:51 smithi029 ceph-mon[29331]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-03-18T01:15:52.069 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:15:51 smithi029 ceph-mon[29331]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-18T01:15:52.069 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:15:51 smithi029 ceph-mon[29331]: osdmap e24: 4 total, 3 up, 4 in 2024-03-18T01:15:52.069 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:15:51 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-18T01:15:52.070 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:15:51 smithi029 ceph-mon[29331]: from='osd.3 [v2:172.21.15.90:6800/1797049062,v1:172.21.15.90:6801/1797049062]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi090", "root=default"]}]: dispatch 2024-03-18T01:15:52.070 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:15:51 smithi029 ceph-mon[29331]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi090", "root=default"]}]: dispatch 2024-03-18T01:15:52.141 INFO:journalctl@ceph.osd.3.smithi090.stdout:Mar 18 01:15:52 smithi090 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-3[42540]: 2024-03-18T01:15:52.086+0000 7fa172d1e700 -1 osd.3 0 waiting for initial osdmap 2024-03-18T01:15:52.141 INFO:journalctl@ceph.osd.3.smithi090.stdout:Mar 18 01:15:52 smithi090 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-3[42540]: 2024-03-18T01:15:52.092+0000 7fa16db19700 -1 osd.3 25 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-18T01:15:53.090 INFO:teuthology.orchestra.run.smithi090.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-18T01:15:53.090 INFO:teuthology.orchestra.run.smithi090.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-18T01:15:53.090 INFO:teuthology.orchestra.run.smithi090.stderr: stderr: 10+0 records in 2024-03-18T01:15:53.091 INFO:teuthology.orchestra.run.smithi090.stderr:10+0 records out 2024-03-18T01:15:53.091 INFO:teuthology.orchestra.run.smithi090.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0126227 s, 831 MB/s 2024-03-18T01:15:53.091 INFO:teuthology.orchestra.run.smithi090.stderr:--> Zapping successful for: 2024-03-18T01:15:53.446 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:15:53 smithi090 ceph-mon[36989]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi090", "root=default"]}]': finished 2024-03-18T01:15:53.446 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:15:53 smithi090 ceph-mon[36989]: osdmap e25: 4 total, 3 up, 4 in 2024-03-18T01:15:53.446 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:15:53 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-18T01:15:53.446 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:15:53 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-18T01:15:53.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:15:53 smithi029 ceph-mon[29331]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi090", "root=default"]}]': finished 2024-03-18T01:15:53.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:15:53 smithi029 ceph-mon[29331]: osdmap e25: 4 total, 3 up, 4 in 2024-03-18T01:15:53.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:15:53 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-18T01:15:53.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:15:53 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-18T01:15:53.730 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 37f51182-e4c4-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi090:vg_nvme/lv_3 2024-03-18T01:15:54.241 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/37f51182-e4c4-11ee-95c9-87774f69a715/mon.smithi090/config 2024-03-18T01:15:54.275 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:15:54 smithi090 ceph-mon[36989]: purged_snaps scrub starts 2024-03-18T01:15:54.276 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:15:54 smithi090 ceph-mon[36989]: purged_snaps scrub ok 2024-03-18T01:15:54.276 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:15:54 smithi090 ceph-mon[36989]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-03-18T01:15:54.276 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:15:54 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-18T01:15:54.276 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:15:54 smithi090 ceph-mon[36989]: osd.3 [v2:172.21.15.90:6800/1797049062,v1:172.21.15.90:6801/1797049062] boot 2024-03-18T01:15:54.276 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:15:54 smithi090 ceph-mon[36989]: osdmap e26: 4 total, 4 up, 4 in 2024-03-18T01:15:54.276 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:15:54 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-18T01:15:54.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:15:54 smithi029 ceph-mon[29331]: purged_snaps scrub starts 2024-03-18T01:15:54.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:15:54 smithi029 ceph-mon[29331]: purged_snaps scrub ok 2024-03-18T01:15:54.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:15:54 smithi029 ceph-mon[29331]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-03-18T01:15:54.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:15:54 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-18T01:15:54.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:15:54 smithi029 ceph-mon[29331]: osd.3 [v2:172.21.15.90:6800/1797049062,v1:172.21.15.90:6801/1797049062] boot 2024-03-18T01:15:54.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:15:54 smithi029 ceph-mon[29331]: osdmap e26: 4 total, 4 up, 4 in 2024-03-18T01:15:54.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:15:54 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-18T01:15:55.430 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:15:55 smithi090 ceph-mon[36989]: osdmap e27: 4 total, 4 up, 4 in 2024-03-18T01:15:55.430 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:15:55 smithi090 ceph-mon[36989]: osdmap e28: 4 total, 4 up, 4 in 2024-03-18T01:15:55.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:15:55 smithi029 ceph-mon[29331]: osdmap e27: 4 total, 4 up, 4 in 2024-03-18T01:15:55.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:15:55 smithi029 ceph-mon[29331]: osdmap e28: 4 total, 4 up, 4 in 2024-03-18T01:15:56.160 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:15:56 smithi090 ceph-mon[36989]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-03-18T01:15:56.160 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:15:56 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:15:56.160 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:15:56 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:15:56.160 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:15:56 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-18T01:15:56.160 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:15:56 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:15:56.160 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:15:56 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:15:56.160 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:15:56 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T01:15:56.160 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:15:56 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:15:56.160 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:15:56 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T01:15:56.160 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:15:56 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:15:56.160 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:15:56 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T01:15:56.160 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:15:56 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:15:56.161 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:15:56 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-18T01:15:56.363 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:15:56.361+0000 7f7d9aba4700 1 -- 172.21.15.90:0/1166801833 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7d940fce70 msgr2=0x7f7d940fd280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:15:56.363 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:15:56.361+0000 7f7d9aba4700 1 --2- 172.21.15.90:0/1166801833 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7d940fce70 0x7f7d940fd280 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f7d88004940 tx=0x7f7d88009e30 comp rx=0 tx=0).stop 2024-03-18T01:15:56.363 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:15:56.361+0000 7f7d9aba4700 1 -- 172.21.15.90:0/1166801833 shutdown_connections 2024-03-18T01:15:56.363 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:15:56.361+0000 7f7d9aba4700 1 --2- 172.21.15.90:0/1166801833 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7d940fdfe0 0x7f7d940fe450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:15:56.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:15:56.361+0000 7f7d9aba4700 1 --2- 172.21.15.90:0/1166801833 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7d940fce70 0x7f7d940fd280 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:15:56.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:15:56.361+0000 7f7d9aba4700 1 -- 172.21.15.90:0/1166801833 >> 172.21.15.90:0/1166801833 conn(0x7f7d940f8550 msgr2=0x7f7d940fa960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:15:56.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:15:56.361+0000 7f7d9aba4700 1 -- 172.21.15.90:0/1166801833 shutdown_connections 2024-03-18T01:15:56.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:15:56.361+0000 7f7d9aba4700 1 -- 172.21.15.90:0/1166801833 wait complete. 2024-03-18T01:15:56.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:15:56.362+0000 7f7d9aba4700 1 Processor -- start 2024-03-18T01:15:56.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:15:56.362+0000 7f7d9aba4700 1 -- start start 2024-03-18T01:15:56.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:15:56.362+0000 7f7d9aba4700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7d940fce70 0x7f7d9410d940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:15:56.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:15:56.362+0000 7f7d9aba4700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7d940fdfe0 0x7f7d9410de80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:15:56.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:15:56.362+0000 7f7d9aba4700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d9410e510 con 0x7f7d940fce70 2024-03-18T01:15:56.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:15:56.362+0000 7f7d9aba4700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d94112eb0 con 0x7f7d940fdfe0 2024-03-18T01:15:56.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:15:56.362+0000 7f7d93fff700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7d940fdfe0 0x7f7d9410de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:15:56.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:15:56.362+0000 7f7d93fff700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7d940fdfe0 0x7f7d9410de80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:45978/0 (socket says 172.21.15.90:45978) 2024-03-18T01:15:56.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:15:56.362+0000 7f7d93fff700 1 -- 172.21.15.90:0/1395997115 learned_addr learned my addr 172.21.15.90:0/1395997115 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-18T01:15:56.365 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:15:56.362+0000 7f7d98940700 1 --2- 172.21.15.90:0/1395997115 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7d940fce70 0x7f7d9410d940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:15:56.365 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:15:56.363+0000 7f7d93fff700 1 -- 172.21.15.90:0/1395997115 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7d940fce70 msgr2=0x7f7d9410d940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:15:56.365 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:15:56.363+0000 7f7d93fff700 1 --2- 172.21.15.90:0/1395997115 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7d940fce70 0x7f7d9410d940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:15:56.365 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:15:56.363+0000 7f7d93fff700 1 -- 172.21.15.90:0/1395997115 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d88005040 con 0x7f7d940fdfe0 2024-03-18T01:15:56.365 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:15:56.363+0000 7f7d93fff700 1 --2- 172.21.15.90:0/1395997115 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7d940fdfe0 0x7f7d9410de80 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f7d8000b790 tx=0x7f7d80009250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:15:56.365 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:15:56.363+0000 7f7d917fa700 1 -- 172.21.15.90:0/1395997115 <== mon.1 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7d8000f770 con 0x7f7d940fdfe0 2024-03-18T01:15:56.365 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:15:56.363+0000 7f7d9aba4700 1 -- 172.21.15.90:0/1395997115 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d94113110 con 0x7f7d940fdfe0 2024-03-18T01:15:56.365 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:15:56.363+0000 7f7d917fa700 1 -- 172.21.15.90:0/1395997115 <== mon.1 v2:172.21.15.90:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7d8000fda0 con 0x7f7d940fdfe0 2024-03-18T01:15:56.365 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:15:56.363+0000 7f7d917fa700 1 -- 172.21.15.90:0/1395997115 <== mon.1 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7d8000d310 con 0x7f7d940fdfe0 2024-03-18T01:15:56.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:15:56.363+0000 7f7d9aba4700 1 -- 172.21.15.90:0/1395997115 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d94113680 con 0x7f7d940fdfe0 2024-03-18T01:15:56.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:15:56.364+0000 7f7d9aba4700 1 -- 172.21.15.90:0/1395997115 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d9404c710 con 0x7f7d940fdfe0 2024-03-18T01:15:56.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:15:56.364+0000 7f7d917fa700 1 -- 172.21.15.90:0/1395997115 <== mon.1 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f7d80010420 con 0x7f7d940fdfe0 2024-03-18T01:15:56.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:15:56.365+0000 7f7d917fa700 1 --2- 172.21.15.90:0/1395997115 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f7d840692a0 0x7f7d8406b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:15:56.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:15:56.365+0000 7f7d917fa700 1 -- 172.21.15.90:0/1395997115 <== mon.1 v2:172.21.15.90:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 3770+0+0 (secure 0 0 0) 0x7f7d80086b10 con 0x7f7d940fdfe0 2024-03-18T01:15:56.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:15:56.365+0000 7f7d98940700 1 --2- 172.21.15.90:0/1395997115 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f7d840692a0 0x7f7d8406b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:15:56.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:15:56.365+0000 7f7d98940700 1 --2- 172.21.15.90:0/1395997115 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f7d840692a0 0x7f7d8406b750 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f7d8800b3b0 tx=0x7f7d88009e30 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:15:56.369 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:15:56.369+0000 7f7d917fa700 1 -- 172.21.15.90:0/1395997115 <== mon.1 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175520 (secure 0 0 0) 0x7f7d80056f70 con 0x7f7d940fdfe0 2024-03-18T01:15:56.558 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:15:56.558+0000 7f7d9aba4700 1 -- 172.21.15.90:0/1395997115 --> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi090:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7f7d94102280 con 0x7f7d840692a0 2024-03-18T01:15:56.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:15:56 smithi029 ceph-mon[29331]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-03-18T01:15:56.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:15:56 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:15:56.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:15:56 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:15:56.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:15:56 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-18T01:15:56.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:15:56 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:15:56.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:15:56 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:15:56.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:15:56 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T01:15:56.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:15:56 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:15:56.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:15:56 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T01:15:56.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:15:56 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:15:56.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:15:56 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T01:15:56.570 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:15:56 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:15:56.570 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:15:56 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-18T01:15:57.243 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:15:57 smithi090 ceph-mon[36989]: Detected new or changed devices on smithi090 2024-03-18T01:15:57.243 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:15:57 smithi090 ceph-mon[36989]: Adjusting osd_memory_target on smithi090 to 15936M 2024-03-18T01:15:57.243 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:15:57 smithi090 ceph-mon[36989]: osdmap e29: 4 total, 4 up, 4 in 2024-03-18T01:15:57.244 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:15:57 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T01:15:57.244 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:15:57 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T01:15:57.244 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:15:57 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:15:57.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:15:57 smithi029 ceph-mon[29331]: Detected new or changed devices on smithi090 2024-03-18T01:15:57.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:15:57 smithi029 ceph-mon[29331]: Adjusting osd_memory_target on smithi090 to 15936M 2024-03-18T01:15:57.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:15:57 smithi029 ceph-mon[29331]: osdmap e29: 4 total, 4 up, 4 in 2024-03-18T01:15:57.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:15:57 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T01:15:57.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:15:57 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T01:15:57.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:15:57 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:15:58.113 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:15:58 smithi090 ceph-mon[36989]: from='client.24155 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi090:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T01:15:58.113 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:15:58 smithi090 ceph-mon[36989]: pgmap v100: 1 pgs: 1 peering; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-18T01:15:58.113 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:15:58 smithi090 ceph-mon[36989]: osdmap e30: 4 total, 4 up, 4 in 2024-03-18T01:15:58.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:15:58 smithi029 ceph-mon[29331]: from='client.24155 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi090:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T01:15:58.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:15:58 smithi029 ceph-mon[29331]: pgmap v100: 1 pgs: 1 peering; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-18T01:15:58.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:15:58 smithi029 ceph-mon[29331]: osdmap e30: 4 total, 4 up, 4 in 2024-03-18T01:15:59.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:15:59 smithi090 ceph-mon[36989]: from='client.? 172.21.15.90:0/3709375621' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c18771da-2efb-4897-a07a-ce48c94e2dee"}]: dispatch 2024-03-18T01:15:59.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:15:59 smithi090 ceph-mon[36989]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c18771da-2efb-4897-a07a-ce48c94e2dee"}]: dispatch 2024-03-18T01:15:59.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:15:59 smithi090 ceph-mon[36989]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c18771da-2efb-4897-a07a-ce48c94e2dee"}]': finished 2024-03-18T01:15:59.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:15:59 smithi090 ceph-mon[36989]: osdmap e31: 5 total, 4 up, 5 in 2024-03-18T01:15:59.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:15:59 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-18T01:15:59.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:15:59 smithi029 ceph-mon[29331]: from='client.? 172.21.15.90:0/3709375621' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c18771da-2efb-4897-a07a-ce48c94e2dee"}]: dispatch 2024-03-18T01:15:59.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:15:59 smithi029 ceph-mon[29331]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c18771da-2efb-4897-a07a-ce48c94e2dee"}]: dispatch 2024-03-18T01:15:59.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:15:59 smithi029 ceph-mon[29331]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c18771da-2efb-4897-a07a-ce48c94e2dee"}]': finished 2024-03-18T01:15:59.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:15:59 smithi029 ceph-mon[29331]: osdmap e31: 5 total, 4 up, 5 in 2024-03-18T01:15:59.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:15:59 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-18T01:16:00.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:16:00 smithi090 ceph-mon[36989]: pgmap v103: 1 pgs: 1 peering; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-18T01:16:00.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:16:00 smithi090 ceph-mon[36989]: from='client.? 172.21.15.90:0/238532184' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T01:16:00.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:16:00 smithi029 ceph-mon[29331]: pgmap v103: 1 pgs: 1 peering; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-18T01:16:00.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:16:00 smithi029 ceph-mon[29331]: from='client.? 172.21.15.90:0/238532184' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T01:16:02.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:16:02 smithi090 ceph-mon[36989]: pgmap v104: 1 pgs: 1 peering; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 98 KiB/s, 0 objects/s recovering 2024-03-18T01:16:02.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:16:02 smithi029 ceph-mon[29331]: pgmap v104: 1 pgs: 1 peering; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 98 KiB/s, 0 objects/s recovering 2024-03-18T01:16:04.492 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:16:04 smithi090 ceph-mon[36989]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 84 KiB/s, 0 objects/s recovering 2024-03-18T01:16:04.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:16:04 smithi029 ceph-mon[29331]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 84 KiB/s, 0 objects/s recovering 2024-03-18T01:16:06.275 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:16:06 smithi090 ceph-mon[36989]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-03-18T01:16:06.276 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:16:06 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-18T01:16:06.276 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:16:06 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:16:06.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:16:06 smithi029 ceph-mon[29331]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-03-18T01:16:06.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:16:06 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-18T01:16:06.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:16:06 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:16:07.413 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:16:07 smithi090 ceph-mon[36989]: Deploying daemon osd.4 on smithi090 2024-03-18T01:16:07.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:16:07 smithi029 ceph-mon[29331]: Deploying daemon osd.4 on smithi090 2024-03-18T01:16:08.396 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:16:08 smithi090 ceph-mon[36989]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-03-18T01:16:08.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:16:08 smithi029 ceph-mon[29331]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-03-18T01:16:10.265 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:16:10 smithi090 ceph-mon[36989]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-03-18T01:16:10.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:16:10 smithi029 ceph-mon[29331]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-03-18T01:16:12.069 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:16:11 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:16:12.069 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:16:11 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:16:12.069 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:16:11 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T01:16:12.069 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:16:11 smithi029 ceph-mon[29331]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-03-18T01:16:12.069 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:16:11 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-18T01:16:12.105 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:16:11 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:16:12.105 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:16:11 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:16:12.105 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:16:11 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T01:16:12.105 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:16:11 smithi090 ceph-mon[36989]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-03-18T01:16:12.105 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:16:11 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-18T01:16:14.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:16:14.069+0000 7f7d917fa700 1 -- 172.21.15.90:0/1395997115 <== mgr.14208 v2:172.21.15.29:6800/1641454714 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f7d94102280 con 0x7f7d840692a0 2024-03-18T01:16:14.071 INFO:teuthology.orchestra.run.smithi090.stdout:Created osd(s) 4 on host 'smithi090' 2024-03-18T01:16:14.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:16:14.071+0000 7f7d9aba4700 1 -- 172.21.15.90:0/1395997115 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f7d840692a0 msgr2=0x7f7d8406b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:16:14.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:16:14.071+0000 7f7d9aba4700 1 --2- 172.21.15.90:0/1395997115 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f7d840692a0 0x7f7d8406b750 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f7d8800b3b0 tx=0x7f7d88009e30 comp rx=0 tx=0).stop 2024-03-18T01:16:14.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:16:14.071+0000 7f7d9aba4700 1 -- 172.21.15.90:0/1395997115 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7d940fdfe0 msgr2=0x7f7d9410de80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:16:14.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:16:14.071+0000 7f7d9aba4700 1 --2- 172.21.15.90:0/1395997115 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7d940fdfe0 0x7f7d9410de80 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f7d8000b790 tx=0x7f7d80009250 comp rx=0 tx=0).stop 2024-03-18T01:16:14.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:16:14.071+0000 7f7d9aba4700 1 -- 172.21.15.90:0/1395997115 shutdown_connections 2024-03-18T01:16:14.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:16:14.071+0000 7f7d9aba4700 1 --2- 172.21.15.90:0/1395997115 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f7d840692a0 0x7f7d8406b750 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:16:14.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:16:14.071+0000 7f7d9aba4700 1 --2- 172.21.15.90:0/1395997115 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7d940fce70 0x7f7d9410d940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:16:14.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:16:14.071+0000 7f7d9aba4700 1 --2- 172.21.15.90:0/1395997115 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7d940fdfe0 0x7f7d9410de80 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:16:14.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:16:14.071+0000 7f7d9aba4700 1 -- 172.21.15.90:0/1395997115 >> 172.21.15.90:0/1395997115 conn(0x7f7d940f8550 msgr2=0x7f7d940f9b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:16:14.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:16:14.071+0000 7f7d9aba4700 1 -- 172.21.15.90:0/1395997115 shutdown_connections 2024-03-18T01:16:14.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:16:14.071+0000 7f7d9aba4700 1 -- 172.21.15.90:0/1395997115 wait complete. 2024-03-18T01:16:14.275 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:16:14 smithi090 ceph-mon[36989]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-03-18T01:16:14.319 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:16:14 smithi029 ceph-mon[29331]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-03-18T01:16:14.711 DEBUG:teuthology.orchestra.run.smithi090:osd.4> sudo journalctl -f -n 0 -u ceph-37f51182-e4c4-11ee-95c9-87774f69a715@osd.4.service 2024-03-18T01:16:14.713 INFO:tasks.cephadm:Deploying osd.5 on smithi090 with /dev/vg_nvme/lv_2... 2024-03-18T01:16:14.713 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 37f51182-e4c4-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-18T01:16:15.013 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/37f51182-e4c4-11ee-95c9-87774f69a715/mon.smithi090/config 2024-03-18T01:16:15.319 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:16:15 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:16:15.319 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:16:15 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:16:15.319 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:16:15 smithi029 ceph-mon[29331]: from='osd.4 [v2:172.21.15.90:6808/2675255441,v1:172.21.15.90:6809/2675255441]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-18T01:16:15.319 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:16:15 smithi029 ceph-mon[29331]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-18T01:16:15.342 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:16:15 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:16:15.342 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:16:15 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:16:15.342 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:16:15 smithi090 ceph-mon[36989]: from='osd.4 [v2:172.21.15.90:6808/2675255441,v1:172.21.15.90:6809/2675255441]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-18T01:16:15.342 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:16:15 smithi090 ceph-mon[36989]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-18T01:16:16.257 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:16:16 smithi090 ceph-mon[36989]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-03-18T01:16:16.258 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:16:16 smithi090 ceph-mon[36989]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-18T01:16:16.258 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:16:16 smithi090 ceph-mon[36989]: osdmap e32: 5 total, 4 up, 5 in 2024-03-18T01:16:16.258 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:16:16 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-18T01:16:16.258 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:16:16 smithi090 ceph-mon[36989]: from='osd.4 [v2:172.21.15.90:6808/2675255441,v1:172.21.15.90:6809/2675255441]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi090", "root=default"]}]: dispatch 2024-03-18T01:16:16.258 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:16:16 smithi090 ceph-mon[36989]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi090", "root=default"]}]: dispatch 2024-03-18T01:16:16.258 INFO:journalctl@ceph.osd.4.smithi090.stdout:Mar 18 01:16:16 smithi090 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-4[49160]: 2024-03-18T01:16:16.048+0000 7f286e30b700 -1 osd.4 0 waiting for initial osdmap 2024-03-18T01:16:16.258 INFO:journalctl@ceph.osd.4.smithi090.stdout:Mar 18 01:16:16 smithi090 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-4[49160]: 2024-03-18T01:16:16.056+0000 7f2869106700 -1 osd.4 33 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-18T01:16:16.318 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:16:16 smithi029 ceph-mon[29331]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-03-18T01:16:16.319 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:16:16 smithi029 ceph-mon[29331]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-18T01:16:16.319 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:16:16 smithi029 ceph-mon[29331]: osdmap e32: 5 total, 4 up, 5 in 2024-03-18T01:16:16.319 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:16:16 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-18T01:16:16.319 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:16:16 smithi029 ceph-mon[29331]: from='osd.4 [v2:172.21.15.90:6808/2675255441,v1:172.21.15.90:6809/2675255441]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi090", "root=default"]}]: dispatch 2024-03-18T01:16:16.319 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:16:16 smithi029 ceph-mon[29331]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi090", "root=default"]}]: dispatch 2024-03-18T01:16:16.815 INFO:teuthology.orchestra.run.smithi090.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-18T01:16:16.815 INFO:teuthology.orchestra.run.smithi090.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-18T01:16:16.815 INFO:teuthology.orchestra.run.smithi090.stderr: stderr: 10+0 records in 2024-03-18T01:16:16.816 INFO:teuthology.orchestra.run.smithi090.stderr:10+0 records out 2024-03-18T01:16:16.816 INFO:teuthology.orchestra.run.smithi090.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0145283 s, 722 MB/s 2024-03-18T01:16:16.816 INFO:teuthology.orchestra.run.smithi090.stderr:--> Zapping successful for: 2024-03-18T01:16:17.276 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:16:17 smithi090 ceph-mon[36989]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi090", "root=default"]}]': finished 2024-03-18T01:16:17.276 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:16:17 smithi090 ceph-mon[36989]: osdmap e33: 5 total, 4 up, 5 in 2024-03-18T01:16:17.276 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:16:17 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-18T01:16:17.276 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:16:17 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-18T01:16:17.276 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:16:17 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:16:17.276 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:16:17 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:16:17.319 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:16:17 smithi029 ceph-mon[29331]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi090", "root=default"]}]': finished 2024-03-18T01:16:17.319 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:16:17 smithi029 ceph-mon[29331]: osdmap e33: 5 total, 4 up, 5 in 2024-03-18T01:16:17.319 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:16:17 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-18T01:16:17.319 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:16:17 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-18T01:16:17.319 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:16:17 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:16:17.319 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:16:17 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:16:17.800 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 37f51182-e4c4-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi090:vg_nvme/lv_2 2024-03-18T01:16:18.268 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/37f51182-e4c4-11ee-95c9-87774f69a715/mon.smithi090/config 2024-03-18T01:16:18.319 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:16:18 smithi029 ceph-mon[29331]: purged_snaps scrub starts 2024-03-18T01:16:18.319 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:16:18 smithi029 ceph-mon[29331]: purged_snaps scrub ok 2024-03-18T01:16:18.319 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:16:18 smithi029 ceph-mon[29331]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-03-18T01:16:18.319 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:16:18 smithi029 ceph-mon[29331]: osd.4 [v2:172.21.15.90:6808/2675255441,v1:172.21.15.90:6809/2675255441] boot 2024-03-18T01:16:18.319 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:16:18 smithi029 ceph-mon[29331]: osdmap e34: 5 total, 5 up, 5 in 2024-03-18T01:16:18.319 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:16:18 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-18T01:16:18.354 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:16:18 smithi090 ceph-mon[36989]: purged_snaps scrub starts 2024-03-18T01:16:18.354 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:16:18 smithi090 ceph-mon[36989]: purged_snaps scrub ok 2024-03-18T01:16:18.354 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:16:18 smithi090 ceph-mon[36989]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-03-18T01:16:18.354 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:16:18 smithi090 ceph-mon[36989]: osd.4 [v2:172.21.15.90:6808/2675255441,v1:172.21.15.90:6809/2675255441] boot 2024-03-18T01:16:18.354 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:16:18 smithi090 ceph-mon[36989]: osdmap e34: 5 total, 5 up, 5 in 2024-03-18T01:16:18.354 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:16:18 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-18T01:16:19.355 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:16:19 smithi090 ceph-mon[36989]: osdmap e35: 5 total, 5 up, 5 in 2024-03-18T01:16:19.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:16:19 smithi029 ceph-mon[29331]: osdmap e35: 5 total, 5 up, 5 in 2024-03-18T01:16:20.122 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:16:20.120+0000 7f46db796700 1 -- 172.21.15.90:0/3240455348 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f46d40fff00 msgr2=0x7f46d4102320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:16:20.122 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:16:20.120+0000 7f46db796700 1 --2- 172.21.15.90:0/3240455348 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f46d40fff00 0x7f46d4102320 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f46c4004940 tx=0x7f46c4009e30 comp rx=0 tx=0).stop 2024-03-18T01:16:20.122 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:16:20.120+0000 7f46db796700 1 -- 172.21.15.90:0/3240455348 shutdown_connections 2024-03-18T01:16:20.122 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:16:20.120+0000 7f46db796700 1 --2- 172.21.15.90:0/3240455348 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f46d40fff00 0x7f46d4102320 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:16:20.122 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:16:20.120+0000 7f46db796700 1 --2- 172.21.15.90:0/3240455348 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f46d40662e0 0x7f46d40ff9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:16:20.122 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:16:20.120+0000 7f46db796700 1 -- 172.21.15.90:0/3240455348 >> 172.21.15.90:0/3240455348 conn(0x7f46d40f7430 msgr2=0x7f46d40f9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:16:20.122 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:16:20.120+0000 7f46db796700 1 -- 172.21.15.90:0/3240455348 shutdown_connections 2024-03-18T01:16:20.122 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:16:20.120+0000 7f46db796700 1 -- 172.21.15.90:0/3240455348 wait complete. 2024-03-18T01:16:20.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:16:20.120+0000 7f46db796700 1 Processor -- start 2024-03-18T01:16:20.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:16:20.120+0000 7f46db796700 1 -- start start 2024-03-18T01:16:20.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:16:20.121+0000 7f46db796700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f46d40662e0 0x7f46d40fd3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:16:20.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:16:20.121+0000 7f46db796700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f46d40fff00 0x7f46d40fb8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:16:20.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:16:20.121+0000 7f46db796700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46d40fbe70 con 0x7f46d40fff00 2024-03-18T01:16:20.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:16:20.121+0000 7f46db796700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46d40fbfe0 con 0x7f46d40662e0 2024-03-18T01:16:20.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:16:20.121+0000 7f46d9532700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f46d40662e0 0x7f46d40fd3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:16:20.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:16:20.121+0000 7f46d9532700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f46d40662e0 0x7f46d40fd3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:60188/0 (socket says 172.21.15.90:60188) 2024-03-18T01:16:20.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:16:20.121+0000 7f46d9532700 1 -- 172.21.15.90:0/4187762099 learned_addr learned my addr 172.21.15.90:0/4187762099 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-18T01:16:20.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:16:20.121+0000 7f46d8d31700 1 --2- 172.21.15.90:0/4187762099 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f46d40fff00 0x7f46d40fb8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:16:20.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:16:20.121+0000 7f46d9532700 1 -- 172.21.15.90:0/4187762099 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f46d40fff00 msgr2=0x7f46d40fb8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:16:20.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:16:20.121+0000 7f46d9532700 1 --2- 172.21.15.90:0/4187762099 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f46d40fff00 0x7f46d40fb8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:16:20.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:16:20.121+0000 7f46d9532700 1 -- 172.21.15.90:0/4187762099 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46c4005040 con 0x7f46d40662e0 2024-03-18T01:16:20.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:16:20.121+0000 7f46d8d31700 1 --2- 172.21.15.90:0/4187762099 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f46d40fff00 0x7f46d40fb8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T01:16:20.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:16:20.121+0000 7f46d9532700 1 --2- 172.21.15.90:0/4187762099 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f46d40662e0 0x7f46d40fd3f0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f46d0002a50 tx=0x7f46d0004600 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:16:20.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:16:20.121+0000 7f46c9ffb700 1 -- 172.21.15.90:0/4187762099 <== mon.1 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f46d0007cc0 con 0x7f46d40662e0 2024-03-18T01:16:20.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:16:20.121+0000 7f46c9ffb700 1 -- 172.21.15.90:0/4187762099 <== mon.1 v2:172.21.15.90:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f46d000f040 con 0x7f46d40662e0 2024-03-18T01:16:20.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:16:20.121+0000 7f46c9ffb700 1 -- 172.21.15.90:0/4187762099 <== mon.1 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f46d000da40 con 0x7f46d40662e0 2024-03-18T01:16:20.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:16:20.121+0000 7f46db796700 1 -- 172.21.15.90:0/4187762099 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46d40fc2c0 con 0x7f46d40662e0 2024-03-18T01:16:20.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:16:20.122+0000 7f46db796700 1 -- 172.21.15.90:0/4187762099 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46d4115a80 con 0x7f46d40662e0 2024-03-18T01:16:20.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:16:20.123+0000 7f46db796700 1 -- 172.21.15.90:0/4187762099 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46d405f5f0 con 0x7f46d40662e0 2024-03-18T01:16:20.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:16:20.123+0000 7f46c9ffb700 1 -- 172.21.15.90:0/4187762099 <== mon.1 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f46d00097c0 con 0x7f46d40662e0 2024-03-18T01:16:20.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:16:20.124+0000 7f46c9ffb700 1 --2- 172.21.15.90:0/4187762099 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f46c00692a0 0x7f46c006b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:16:20.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:16:20.124+0000 7f46c9ffb700 1 -- 172.21.15.90:0/4187762099 <== mon.1 v2:172.21.15.90:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 4202+0+0 (secure 0 0 0) 0x7f46d00869f0 con 0x7f46d40662e0 2024-03-18T01:16:20.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:16:20.124+0000 7f46d8d31700 1 --2- 172.21.15.90:0/4187762099 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f46c00692a0 0x7f46c006b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:16:20.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:16:20.125+0000 7f46d8d31700 1 --2- 172.21.15.90:0/4187762099 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f46c00692a0 0x7f46c006b750 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f46c400b3b0 tx=0x7f46c4009e30 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:16:20.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:16:20.131+0000 7f46c9ffb700 1 -- 172.21.15.90:0/4187762099 <== mon.1 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175520 (secure 0 0 0) 0x7f46d0088e60 con 0x7f46d40662e0 2024-03-18T01:16:20.276 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:16:20 smithi090 ceph-mon[36989]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-03-18T01:16:20.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:16:20.376+0000 7f46db796700 1 -- 172.21.15.90:0/4187762099 --> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi090:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7f46d40f8f70 con 0x7f46c00692a0 2024-03-18T01:16:20.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:16:20 smithi029 ceph-mon[29331]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-03-18T01:16:21.265 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:16:21 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T01:16:21.265 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:16:21 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T01:16:21.265 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:16:21 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:16:21.265 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:16:21 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:16:21.265 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:16:21 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:16:21.265 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:16:21 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-18T01:16:21.265 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:16:21 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-18T01:16:21.265 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:16:21 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:16:21.265 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:16:21 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:16:21.265 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:16:21 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T01:16:21.265 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:16:21 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:16:21.265 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:16:21 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T01:16:21.265 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:16:21 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:16:21.266 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:16:21 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T01:16:21.266 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:16:21 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:16:21.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:16:21 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T01:16:21.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:16:21 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T01:16:21.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:16:21 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:16:21.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:16:21 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:16:21.570 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:16:21 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:16:21.570 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:16:21 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-18T01:16:21.570 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:16:21 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-18T01:16:21.570 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:16:21 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:16:21.570 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:16:21 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:16:21.570 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:16:21 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T01:16:21.570 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:16:21 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:16:21.570 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:16:21 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T01:16:21.570 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:16:21 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:16:21.570 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:16:21 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T01:16:21.570 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:16:21 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:16:22.395 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:16:22 smithi090 ceph-mon[36989]: from='client.24173 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi090:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T01:16:22.395 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:16:22 smithi090 ceph-mon[36989]: Detected new or changed devices on smithi090 2024-03-18T01:16:22.395 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:16:22 smithi090 ceph-mon[36989]: Adjusting osd_memory_target on smithi090 to 7968M 2024-03-18T01:16:22.395 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:16:22 smithi090 ceph-mon[36989]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-03-18T01:16:22.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:16:22 smithi029 ceph-mon[29331]: from='client.24173 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi090:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T01:16:22.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:16:22 smithi029 ceph-mon[29331]: Detected new or changed devices on smithi090 2024-03-18T01:16:22.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:16:22 smithi029 ceph-mon[29331]: Adjusting osd_memory_target on smithi090 to 7968M 2024-03-18T01:16:22.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:16:22 smithi029 ceph-mon[29331]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-03-18T01:16:23.459 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:16:23 smithi090 ceph-mon[36989]: from='client.? 172.21.15.90:0/757003653' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "82248e69-74f9-4f46-a3fd-9224dcc8eae6"}]: dispatch 2024-03-18T01:16:23.459 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:16:23 smithi090 ceph-mon[36989]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "82248e69-74f9-4f46-a3fd-9224dcc8eae6"}]: dispatch 2024-03-18T01:16:23.459 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:16:23 smithi090 ceph-mon[36989]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "82248e69-74f9-4f46-a3fd-9224dcc8eae6"}]': finished 2024-03-18T01:16:23.459 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:16:23 smithi090 ceph-mon[36989]: osdmap e36: 6 total, 5 up, 6 in 2024-03-18T01:16:23.459 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:16:23 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-18T01:16:23.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:16:23 smithi029 ceph-mon[29331]: from='client.? 172.21.15.90:0/757003653' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "82248e69-74f9-4f46-a3fd-9224dcc8eae6"}]: dispatch 2024-03-18T01:16:23.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:16:23 smithi029 ceph-mon[29331]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "82248e69-74f9-4f46-a3fd-9224dcc8eae6"}]: dispatch 2024-03-18T01:16:23.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:16:23 smithi029 ceph-mon[29331]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "82248e69-74f9-4f46-a3fd-9224dcc8eae6"}]': finished 2024-03-18T01:16:23.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:16:23 smithi029 ceph-mon[29331]: osdmap e36: 6 total, 5 up, 6 in 2024-03-18T01:16:23.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:16:23 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-18T01:16:24.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:16:24 smithi090 ceph-mon[36989]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-03-18T01:16:24.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:16:24 smithi090 ceph-mon[36989]: from='client.? 172.21.15.90:0/1602532975' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T01:16:24.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:16:24 smithi029 ceph-mon[29331]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-03-18T01:16:24.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:16:24 smithi029 ceph-mon[29331]: from='client.? 172.21.15.90:0/1602532975' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T01:16:26.319 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:16:26 smithi029 ceph-mon[29331]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-03-18T01:16:26.319 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:16:26 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-18T01:16:26.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:16:26 smithi090 ceph-mon[36989]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-03-18T01:16:26.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:16:26 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-18T01:16:28.276 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:16:28 smithi090 ceph-mon[36989]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-03-18T01:16:28.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:16:28 smithi029 ceph-mon[29331]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-03-18T01:16:30.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:16:30 smithi090 ceph-mon[36989]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-03-18T01:16:30.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:16:30 smithi029 ceph-mon[29331]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-03-18T01:16:31.276 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:16:31 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-18T01:16:31.276 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:16:31 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:16:31.276 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:16:31 smithi090 ceph-mon[36989]: Deploying daemon osd.5 on smithi090 2024-03-18T01:16:31.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:16:31 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-18T01:16:31.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:16:31 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:16:31.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:16:31 smithi029 ceph-mon[29331]: Deploying daemon osd.5 on smithi090 2024-03-18T01:16:32.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:16:32 smithi090 ceph-mon[36989]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-03-18T01:16:32.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:16:32 smithi029 ceph-mon[29331]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-03-18T01:16:34.404 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:16:34 smithi090 ceph-mon[36989]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-03-18T01:16:34.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:16:34 smithi029 ceph-mon[29331]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-03-18T01:16:36.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:16:36 smithi090 ceph-mon[36989]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-03-18T01:16:36.543 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:16:36 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:16:36.543 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:16:36 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:16:36.543 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:16:36 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T01:16:36.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:16:36 smithi029 ceph-mon[29331]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-03-18T01:16:36.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:16:36 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:16:36.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:16:36 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:16:36.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:16:36 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T01:16:38.500 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:16:38 smithi090 ceph-mon[36989]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-03-18T01:16:38.543 INFO:teuthology.orchestra.run.smithi090.stdout:Created osd(s) 5 on host 'smithi090' 2024-03-18T01:16:38.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:16:38.542+0000 7f46c9ffb700 1 -- 172.21.15.90:0/4187762099 <== mgr.14208 v2:172.21.15.29:6800/1641454714 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f46d40f8f70 con 0x7f46c00692a0 2024-03-18T01:16:38.545 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:16:38.545+0000 7f46db796700 1 -- 172.21.15.90:0/4187762099 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f46c00692a0 msgr2=0x7f46c006b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:16:38.545 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:16:38.545+0000 7f46db796700 1 --2- 172.21.15.90:0/4187762099 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f46c00692a0 0x7f46c006b750 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f46c400b3b0 tx=0x7f46c4009e30 comp rx=0 tx=0).stop 2024-03-18T01:16:38.545 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:16:38.545+0000 7f46db796700 1 -- 172.21.15.90:0/4187762099 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f46d40662e0 msgr2=0x7f46d40fd3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:16:38.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:16:38.545+0000 7f46db796700 1 --2- 172.21.15.90:0/4187762099 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f46d40662e0 0x7f46d40fd3f0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f46d0002a50 tx=0x7f46d0004600 comp rx=0 tx=0).stop 2024-03-18T01:16:38.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:16:38.545+0000 7f46db796700 1 -- 172.21.15.90:0/4187762099 shutdown_connections 2024-03-18T01:16:38.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:16:38.545+0000 7f46db796700 1 --2- 172.21.15.90:0/4187762099 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f46c00692a0 0x7f46c006b750 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:16:38.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:16:38.545+0000 7f46db796700 1 --2- 172.21.15.90:0/4187762099 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f46d40662e0 0x7f46d40fd3f0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:16:38.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:16:38.545+0000 7f46db796700 1 --2- 172.21.15.90:0/4187762099 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f46d40fff00 0x7f46d40fb8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:16:38.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:16:38.545+0000 7f46db796700 1 -- 172.21.15.90:0/4187762099 >> 172.21.15.90:0/4187762099 conn(0x7f46d40f7430 msgr2=0x7f46d40f7850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:16:38.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:16:38.545+0000 7f46db796700 1 -- 172.21.15.90:0/4187762099 shutdown_connections 2024-03-18T01:16:38.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:16:38.545+0000 7f46db796700 1 -- 172.21.15.90:0/4187762099 wait complete. 2024-03-18T01:16:38.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:16:38 smithi029 ceph-mon[29331]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-03-18T01:16:39.372 DEBUG:teuthology.orchestra.run.smithi090:osd.5> sudo journalctl -f -n 0 -u ceph-37f51182-e4c4-11ee-95c9-87774f69a715@osd.5.service 2024-03-18T01:16:39.374 INFO:tasks.cephadm:Waiting for 6 OSDs to come up... 2024-03-18T01:16:39.374 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 37f51182-e4c4-11ee-95c9-87774f69a715 -- ceph osd stat -f json 2024-03-18T01:16:39.653 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/37f51182-e4c4-11ee-95c9-87774f69a715/mon.smithi029/config 2024-03-18T01:16:39.775 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:16:39 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:16:39.775 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:16:39 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:16:39.775 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:16:39 smithi029 ceph-mon[29331]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-03-18T01:16:39.775 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:16:39 smithi029 ceph-mon[29331]: from='osd.5 [v2:172.21.15.90:6816/2630668390,v1:172.21.15.90:6817/2630668390]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-18T01:16:39.775 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:16:39 smithi029 ceph-mon[29331]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-18T01:16:39.776 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:16:39 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:16:39.776 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:16:39 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:16:39.776 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:16:39 smithi090 ceph-mon[36989]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-03-18T01:16:39.776 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:16:39 smithi090 ceph-mon[36989]: from='osd.5 [v2:172.21.15.90:6816/2630668390,v1:172.21.15.90:6817/2630668390]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-18T01:16:39.776 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:16:39 smithi090 ceph-mon[36989]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-18T01:16:40.654 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:16:40 smithi090 ceph-mon[36989]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-18T01:16:40.654 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:16:40 smithi090 ceph-mon[36989]: osdmap e37: 6 total, 5 up, 6 in 2024-03-18T01:16:40.655 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:16:40 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-18T01:16:40.655 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:16:40 smithi090 ceph-mon[36989]: from='osd.5 [v2:172.21.15.90:6816/2630668390,v1:172.21.15.90:6817/2630668390]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi090", "root=default"]}]: dispatch 2024-03-18T01:16:40.655 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:16:40 smithi090 ceph-mon[36989]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi090", "root=default"]}]: dispatch 2024-03-18T01:16:40.655 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:16:40 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:16:40.655 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:16:40 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:16:40.655 INFO:journalctl@ceph.osd.5.smithi090.stdout:Mar 18 01:16:40 smithi090 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-5[55599]: 2024-03-18T01:16:40.542+0000 7ffbd8ffe700 -1 osd.5 0 waiting for initial osdmap 2024-03-18T01:16:40.655 INFO:journalctl@ceph.osd.5.smithi090.stdout:Mar 18 01:16:40 smithi090 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-5[55599]: 2024-03-18T01:16:40.550+0000 7ffbd05f2700 -1 osd.5 38 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-18T01:16:40.820 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:16:40 smithi029 ceph-mon[29331]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-18T01:16:40.820 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:16:40 smithi029 ceph-mon[29331]: osdmap e37: 6 total, 5 up, 6 in 2024-03-18T01:16:40.820 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:16:40 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-18T01:16:40.820 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:16:40 smithi029 ceph-mon[29331]: from='osd.5 [v2:172.21.15.90:6816/2630668390,v1:172.21.15.90:6817/2630668390]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi090", "root=default"]}]: dispatch 2024-03-18T01:16:40.820 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:16:40 smithi029 ceph-mon[29331]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi090", "root=default"]}]: dispatch 2024-03-18T01:16:40.820 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:16:40 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:16:40.821 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:16:40 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:16:41.325 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:41.323+0000 7f8563e89700 1 -- 172.21.15.29:0/2947814075 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f855c0fb200 msgr2=0x7f855c0fb610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:16:41.325 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:41.323+0000 7f8563e89700 1 --2- 172.21.15.29:0/2947814075 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f855c0fb200 0x7f855c0fb610 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f8558004940 tx=0x7f8558009e30 comp rx=0 tx=0).stop 2024-03-18T01:16:41.325 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:41.323+0000 7f8563e89700 1 -- 172.21.15.29:0/2947814075 shutdown_connections 2024-03-18T01:16:41.325 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:41.323+0000 7f8563e89700 1 --2- 172.21.15.29:0/2947814075 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f855c0fbb50 0x7f855c0fe3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:16:41.326 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:41.323+0000 7f8563e89700 1 --2- 172.21.15.29:0/2947814075 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f855c0fb200 0x7f855c0fb610 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:16:41.326 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:41.323+0000 7f8563e89700 1 -- 172.21.15.29:0/2947814075 >> 172.21.15.29:0/2947814075 conn(0x7f855c0f6be0 msgr2=0x7f855c0f9010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:16:41.326 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:41.323+0000 7f8563e89700 1 -- 172.21.15.29:0/2947814075 shutdown_connections 2024-03-18T01:16:41.326 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:41.323+0000 7f8563e89700 1 -- 172.21.15.29:0/2947814075 wait complete. 2024-03-18T01:16:41.326 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:41.324+0000 7f8563e89700 1 Processor -- start 2024-03-18T01:16:41.326 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:41.324+0000 7f8563e89700 1 -- start start 2024-03-18T01:16:41.326 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:41.324+0000 7f8563e89700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f855c0fb200 0x7f855c10db70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:16:41.326 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:41.324+0000 7f8563e89700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f855c0fbb50 0x7f855c10e0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:16:41.326 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:41.324+0000 7f8563e89700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f855c10e6b0 con 0x7f855c0fb200 2024-03-18T01:16:41.326 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:41.324+0000 7f8563e89700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f855c10e820 con 0x7f855c0fbb50 2024-03-18T01:16:41.326 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:41.324+0000 7f8561c25700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f855c0fb200 0x7f855c10db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:16:41.326 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:41.324+0000 7f8561c25700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f855c0fb200 0x7f855c10db70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57242/0 (socket says 172.21.15.29:57242) 2024-03-18T01:16:41.326 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:41.324+0000 7f8561c25700 1 -- 172.21.15.29:0/1384295623 learned_addr learned my addr 172.21.15.29:0/1384295623 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-18T01:16:41.327 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:41.324+0000 7f8561424700 1 --2- 172.21.15.29:0/1384295623 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f855c0fbb50 0x7f855c10e0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:16:41.327 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:41.325+0000 7f8561c25700 1 -- 172.21.15.29:0/1384295623 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f855c0fbb50 msgr2=0x7f855c10e0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:16:41.327 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:41.325+0000 7f8561c25700 1 --2- 172.21.15.29:0/1384295623 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f855c0fbb50 0x7f855c10e0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:16:41.327 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:41.325+0000 7f8561c25700 1 -- 172.21.15.29:0/1384295623 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8558005040 con 0x7f855c0fb200 2024-03-18T01:16:41.327 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:41.325+0000 7f8561c25700 1 --2- 172.21.15.29:0/1384295623 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f855c0fb200 0x7f855c10db70 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f8558009150 tx=0x7f8558009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:16:41.327 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:41.325+0000 7f854e7fc700 1 -- 172.21.15.29:0/1384295623 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f855800e040 con 0x7f855c0fb200 2024-03-18T01:16:41.327 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:41.325+0000 7f854e7fc700 1 -- 172.21.15.29:0/1384295623 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8558004290 con 0x7f855c0fb200 2024-03-18T01:16:41.327 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:41.325+0000 7f8563e89700 1 -- 172.21.15.29:0/1384295623 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f855c113220 con 0x7f855c0fb200 2024-03-18T01:16:41.327 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:41.325+0000 7f854e7fc700 1 -- 172.21.15.29:0/1384295623 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f855801c630 con 0x7f855c0fb200 2024-03-18T01:16:41.327 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:41.325+0000 7f8563e89700 1 -- 172.21.15.29:0/1384295623 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f855c1137e0 con 0x7f855c0fb200 2024-03-18T01:16:41.328 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:41.326+0000 7f8563e89700 1 -- 172.21.15.29:0/1384295623 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f855c04c710 con 0x7f855c0fb200 2024-03-18T01:16:41.328 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:41.326+0000 7f854e7fc700 1 -- 172.21.15.29:0/1384295623 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f855800bbf0 con 0x7f855c0fb200 2024-03-18T01:16:41.328 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:41.327+0000 7f854e7fc700 1 --2- 172.21.15.29:0/1384295623 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f85480692a0 0x7f854806b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:16:41.328 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:41.327+0000 7f854e7fc700 1 -- 172.21.15.29:0/1384295623 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 4354+0+0 (secure 0 0 0) 0x7f8558087920 con 0x7f855c0fb200 2024-03-18T01:16:41.328 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:41.327+0000 7f8561424700 1 --2- 172.21.15.29:0/1384295623 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f85480692a0 0x7f854806b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:16:41.328 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:41.327+0000 7f8561424700 1 --2- 172.21.15.29:0/1384295623 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f85480692a0 0x7f854806b750 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f8550007fd0 tx=0x7f8550009040 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:16:41.332 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:41.331+0000 7f854e7fc700 1 -- 172.21.15.29:0/1384295623 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175520 (secure 0 0 0) 0x7f8558057bd0 con 0x7f855c0fb200 2024-03-18T01:16:41.522 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:41.522+0000 7f8563e89700 1 -- 172.21.15.29:0/1384295623 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) v1 -- 0x7f855c066050 con 0x7f855c0fb200 2024-03-18T01:16:41.522 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:41.522+0000 7f854e7fc700 1 -- 172.21.15.29:0/1384295623 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v38) v1 ==== 74+0+130 (secure 0 0 0) 0x7f8558057760 con 0x7f855c0fb200 2024-03-18T01:16:41.523 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-18T01:16:41.525 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:41.525+0000 7f8563e89700 1 -- 172.21.15.29:0/1384295623 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f85480692a0 msgr2=0x7f854806b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:16:41.526 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:41.525+0000 7f8563e89700 1 --2- 172.21.15.29:0/1384295623 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f85480692a0 0x7f854806b750 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f8550007fd0 tx=0x7f8550009040 comp rx=0 tx=0).stop 2024-03-18T01:16:41.526 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:41.525+0000 7f8563e89700 1 -- 172.21.15.29:0/1384295623 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f855c0fb200 msgr2=0x7f855c10db70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:16:41.526 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:41.525+0000 7f8563e89700 1 --2- 172.21.15.29:0/1384295623 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f855c0fb200 0x7f855c10db70 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f8558009150 tx=0x7f8558009e30 comp rx=0 tx=0).stop 2024-03-18T01:16:41.526 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:41.525+0000 7f8563e89700 1 -- 172.21.15.29:0/1384295623 shutdown_connections 2024-03-18T01:16:41.526 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:41.525+0000 7f8563e89700 1 --2- 172.21.15.29:0/1384295623 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f85480692a0 0x7f854806b750 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:16:41.526 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:41.525+0000 7f8563e89700 1 --2- 172.21.15.29:0/1384295623 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f855c0fb200 0x7f855c10db70 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:16:41.526 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:41.525+0000 7f8563e89700 1 --2- 172.21.15.29:0/1384295623 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f855c0fbb50 0x7f855c10e0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:16:41.526 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:41.525+0000 7f8563e89700 1 -- 172.21.15.29:0/1384295623 >> 172.21.15.29:0/1384295623 conn(0x7f855c0f6be0 msgr2=0x7f855c0f82f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:16:41.526 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:41.525+0000 7f8563e89700 1 -- 172.21.15.29:0/1384295623 shutdown_connections 2024-03-18T01:16:41.526 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:41.525+0000 7f8563e89700 1 -- 172.21.15.29:0/1384295623 wait complete. 2024-03-18T01:16:41.776 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:16:41 smithi090 ceph-mon[36989]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi090", "root=default"]}]': finished 2024-03-18T01:16:41.776 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:16:41 smithi090 ceph-mon[36989]: osdmap e38: 6 total, 5 up, 6 in 2024-03-18T01:16:41.776 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:16:41 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-18T01:16:41.776 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:16:41 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-18T01:16:41.776 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:16:41 smithi090 ceph-mon[36989]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-03-18T01:16:41.776 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:16:41 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-18T01:16:41.776 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:16:41 smithi090 ceph-mon[36989]: from='client.? 172.21.15.29:0/1384295623' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-18T01:16:41.819 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:16:41 smithi029 ceph-mon[29331]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi090", "root=default"]}]': finished 2024-03-18T01:16:41.819 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:16:41 smithi029 ceph-mon[29331]: osdmap e38: 6 total, 5 up, 6 in 2024-03-18T01:16:41.819 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:16:41 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-18T01:16:41.819 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:16:41 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-18T01:16:41.819 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:16:41 smithi029 ceph-mon[29331]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-03-18T01:16:41.819 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:16:41 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-18T01:16:41.819 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:16:41 smithi029 ceph-mon[29331]: from='client.? 172.21.15.29:0/1384295623' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-18T01:16:42.222 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":38,"num_osds":6,"num_up_osds":5,"osd_up_since":1710724577,"num_in_osds":6,"osd_in_since":1710724583,"num_remapped_pgs":0} 2024-03-18T01:16:42.819 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:16:42 smithi029 ceph-mon[29331]: purged_snaps scrub starts 2024-03-18T01:16:42.819 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:16:42 smithi029 ceph-mon[29331]: purged_snaps scrub ok 2024-03-18T01:16:42.819 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:16:42 smithi029 ceph-mon[29331]: osd.5 [v2:172.21.15.90:6816/2630668390,v1:172.21.15.90:6817/2630668390] boot 2024-03-18T01:16:42.819 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:16:42 smithi029 ceph-mon[29331]: osdmap e39: 6 total, 6 up, 6 in 2024-03-18T01:16:42.819 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:16:42 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-18T01:16:42.894 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:16:42 smithi090 ceph-mon[36989]: purged_snaps scrub starts 2024-03-18T01:16:42.895 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:16:42 smithi090 ceph-mon[36989]: purged_snaps scrub ok 2024-03-18T01:16:42.895 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:16:42 smithi090 ceph-mon[36989]: osd.5 [v2:172.21.15.90:6816/2630668390,v1:172.21.15.90:6817/2630668390] boot 2024-03-18T01:16:42.895 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:16:42 smithi090 ceph-mon[36989]: osdmap e39: 6 total, 6 up, 6 in 2024-03-18T01:16:42.895 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:16:42 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-18T01:16:43.224 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 37f51182-e4c4-11ee-95c9-87774f69a715 -- ceph osd stat -f json 2024-03-18T01:16:43.499 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/37f51182-e4c4-11ee-95c9-87774f69a715/mon.smithi029/config 2024-03-18T01:16:43.819 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:16:43 smithi029 ceph-mon[29331]: osdmap e40: 6 total, 6 up, 6 in 2024-03-18T01:16:43.819 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:16:43 smithi029 ceph-mon[29331]: Detected new or changed devices on smithi090 2024-03-18T01:16:43.819 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:16:43 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:16:43.819 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:16:43 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:16:43.819 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:16:43 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-18T01:16:43.819 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:16:43 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-18T01:16:43.819 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:16:43 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-18T01:16:43.819 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:16:43 smithi029 ceph-mon[29331]: Adjusting osd_memory_target on smithi090 to 5312M 2024-03-18T01:16:43.819 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:16:43 smithi029 ceph-mon[29331]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-18T01:16:43.820 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:16:43 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:16:43.820 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:16:43 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:16:43.820 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:16:43 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T01:16:43.820 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:16:43 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:16:43.820 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:16:43 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T01:16:43.820 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:16:43 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:16:43.820 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:16:43 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T01:16:43.820 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:16:43 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:16:44.026 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:16:43 smithi090 ceph-mon[36989]: osdmap e40: 6 total, 6 up, 6 in 2024-03-18T01:16:44.026 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:16:43 smithi090 ceph-mon[36989]: Detected new or changed devices on smithi090 2024-03-18T01:16:44.026 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:16:43 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:16:44.026 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:16:43 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:16:44.026 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:16:43 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-18T01:16:44.026 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:16:43 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-18T01:16:44.026 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:16:43 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-18T01:16:44.026 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:16:43 smithi090 ceph-mon[36989]: Adjusting osd_memory_target on smithi090 to 5312M 2024-03-18T01:16:44.026 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:16:43 smithi090 ceph-mon[36989]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-18T01:16:44.027 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:16:43 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:16:44.027 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:16:43 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:16:44.027 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:16:43 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T01:16:44.027 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:16:43 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:16:44.027 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:16:43 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T01:16:44.027 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:16:43 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:16:44.027 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:16:43 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T01:16:44.027 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:16:43 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:16:45.230 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:45.229+0000 7fe903076700 1 -- 172.21.15.29:0/1737468447 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe8fc0fbd00 msgr2=0x7fe8fc0fc110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:16:45.230 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:45.229+0000 7fe903076700 1 --2- 172.21.15.29:0/1737468447 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe8fc0fbd00 0x7fe8fc0fc110 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fe8f0004940 tx=0x7fe8f0009e30 comp rx=0 tx=0).stop 2024-03-18T01:16:45.230 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:45.229+0000 7fe903076700 1 -- 172.21.15.29:0/1737468447 shutdown_connections 2024-03-18T01:16:45.231 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:45.229+0000 7fe903076700 1 --2- 172.21.15.29:0/1737468447 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe8fc0fcfb0 0x7fe8fc0fd420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:16:45.231 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:45.229+0000 7fe903076700 1 --2- 172.21.15.29:0/1737468447 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe8fc0fbd00 0x7fe8fc0fc110 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:16:45.231 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:45.229+0000 7fe903076700 1 -- 172.21.15.29:0/1737468447 >> 172.21.15.29:0/1737468447 conn(0x7fe8fc0f7480 msgr2=0x7fe8fc0f9890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:16:45.231 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:45.230+0000 7fe903076700 1 -- 172.21.15.29:0/1737468447 shutdown_connections 2024-03-18T01:16:45.231 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:45.230+0000 7fe903076700 1 -- 172.21.15.29:0/1737468447 wait complete. 2024-03-18T01:16:45.231 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:45.230+0000 7fe903076700 1 Processor -- start 2024-03-18T01:16:45.231 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:45.230+0000 7fe903076700 1 -- start start 2024-03-18T01:16:45.232 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:45.231+0000 7fe903076700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe8fc0fbd00 0x7fe8fc10c820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:16:45.233 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:45.231+0000 7fe903076700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe8fc0fcfb0 0x7fe8fc10cd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:16:45.233 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:45.231+0000 7fe903076700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8fc10d360 con 0x7fe8fc0fcfb0 2024-03-18T01:16:45.233 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:45.231+0000 7fe903076700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8fc10d4d0 con 0x7fe8fc0fbd00 2024-03-18T01:16:45.233 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:45.231+0000 7fe8fbfff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe8fc0fcfb0 0x7fe8fc10cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:16:45.233 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:45.231+0000 7fe8fbfff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe8fc0fcfb0 0x7fe8fc10cd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:38152/0 (socket says 172.21.15.29:38152) 2024-03-18T01:16:45.233 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:45.231+0000 7fe8fbfff700 1 -- 172.21.15.29:0/3027740973 learned_addr learned my addr 172.21.15.29:0/3027740973 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-18T01:16:45.233 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:45.231+0000 7fe900e12700 1 --2- 172.21.15.29:0/3027740973 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe8fc0fbd00 0x7fe8fc10c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:16:45.233 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:45.231+0000 7fe8fbfff700 1 -- 172.21.15.29:0/3027740973 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe8fc0fbd00 msgr2=0x7fe8fc10c820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:16:45.233 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:45.231+0000 7fe8fbfff700 1 --2- 172.21.15.29:0/3027740973 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe8fc0fbd00 0x7fe8fc10c820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:16:45.233 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:45.231+0000 7fe8fbfff700 1 -- 172.21.15.29:0/3027740973 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8f0005040 con 0x7fe8fc0fcfb0 2024-03-18T01:16:45.233 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:45.231+0000 7fe8fbfff700 1 --2- 172.21.15.29:0/3027740973 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe8fc0fcfb0 0x7fe8fc10cd60 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fe8e800c830 tx=0x7fe8e8004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:16:45.233 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:45.231+0000 7fe8f97fa700 1 -- 172.21.15.29:0/3027740973 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe8e80106f0 con 0x7fe8fc0fcfb0 2024-03-18T01:16:45.233 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:45.231+0000 7fe8f97fa700 1 -- 172.21.15.29:0/3027740973 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe8e8010d20 con 0x7fe8fc0fcfb0 2024-03-18T01:16:45.234 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:45.231+0000 7fe903076700 1 -- 172.21.15.29:0/3027740973 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8fc111f30 con 0x7fe8fc0fcfb0 2024-03-18T01:16:45.234 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:45.231+0000 7fe8f97fa700 1 -- 172.21.15.29:0/3027740973 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe8e80113a0 con 0x7fe8fc0fcfb0 2024-03-18T01:16:45.234 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:45.231+0000 7fe903076700 1 -- 172.21.15.29:0/3027740973 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8fc1124c0 con 0x7fe8fc0fcfb0 2024-03-18T01:16:45.234 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:45.233+0000 7fe903076700 1 -- 172.21.15.29:0/3027740973 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8fc04c710 con 0x7fe8fc0fcfb0 2024-03-18T01:16:45.235 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:45.233+0000 7fe8f97fa700 1 -- 172.21.15.29:0/3027740973 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7fe8e8012400 con 0x7fe8fc0fcfb0 2024-03-18T01:16:45.235 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:45.233+0000 7fe8f97fa700 1 --2- 172.21.15.29:0/3027740973 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7fe8ec0692a0 0x7fe8ec06b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:16:45.235 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:45.233+0000 7fe8f97fa700 1 -- 172.21.15.29:0/3027740973 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fe8e8086eb0 con 0x7fe8fc0fcfb0 2024-03-18T01:16:45.235 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:45.233+0000 7fe900e12700 1 --2- 172.21.15.29:0/3027740973 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7fe8ec0692a0 0x7fe8ec06b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:16:45.235 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:45.234+0000 7fe900e12700 1 --2- 172.21.15.29:0/3027740973 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7fe8ec0692a0 0x7fe8ec06b750 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fe8f000b3b0 tx=0x7fe8f0009e30 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:16:45.238 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:45.238+0000 7fe8f97fa700 1 -- 172.21.15.29:0/3027740973 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175520 (secure 0 0 0) 0x7fe8e8056f10 con 0x7fe8fc0fcfb0 2024-03-18T01:16:45.432 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:45.431+0000 7fe903076700 1 -- 172.21.15.29:0/3027740973 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) v1 -- 0x7fe8fc1047f0 con 0x7fe8fc0fcfb0 2024-03-18T01:16:45.432 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:45.432+0000 7fe8f97fa700 1 -- 172.21.15.29:0/3027740973 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v40) v1 ==== 74+0+130 (secure 0 0 0) 0x7fe8e8056aa0 con 0x7fe8fc0fcfb0 2024-03-18T01:16:45.432 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-18T01:16:45.434 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:45.434+0000 7fe903076700 1 -- 172.21.15.29:0/3027740973 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7fe8ec0692a0 msgr2=0x7fe8ec06b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:16:45.434 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:45.434+0000 7fe903076700 1 --2- 172.21.15.29:0/3027740973 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7fe8ec0692a0 0x7fe8ec06b750 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fe8f000b3b0 tx=0x7fe8f0009e30 comp rx=0 tx=0).stop 2024-03-18T01:16:45.434 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:45.434+0000 7fe903076700 1 -- 172.21.15.29:0/3027740973 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe8fc0fcfb0 msgr2=0x7fe8fc10cd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:16:45.434 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:45.434+0000 7fe903076700 1 --2- 172.21.15.29:0/3027740973 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe8fc0fcfb0 0x7fe8fc10cd60 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fe8e800c830 tx=0x7fe8e8004470 comp rx=0 tx=0).stop 2024-03-18T01:16:45.434 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:45.434+0000 7fe903076700 1 -- 172.21.15.29:0/3027740973 shutdown_connections 2024-03-18T01:16:45.434 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:45.434+0000 7fe903076700 1 --2- 172.21.15.29:0/3027740973 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7fe8ec0692a0 0x7fe8ec06b750 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:16:45.434 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:45.434+0000 7fe903076700 1 --2- 172.21.15.29:0/3027740973 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe8fc0fbd00 0x7fe8fc10c820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:16:45.434 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:45.434+0000 7fe903076700 1 --2- 172.21.15.29:0/3027740973 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe8fc0fcfb0 0x7fe8fc10cd60 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:16:45.435 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:45.434+0000 7fe903076700 1 -- 172.21.15.29:0/3027740973 >> 172.21.15.29:0/3027740973 conn(0x7fe8fc0f7480 msgr2=0x7fe8fc0f8b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:16:45.435 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:45.434+0000 7fe903076700 1 -- 172.21.15.29:0/3027740973 shutdown_connections 2024-03-18T01:16:45.435 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:45.434+0000 7fe903076700 1 -- 172.21.15.29:0/3027740973 wait complete. 2024-03-18T01:16:46.162 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":40,"num_osds":6,"num_up_osds":6,"osd_up_since":1710724601,"num_in_osds":6,"osd_in_since":1710724583,"num_remapped_pgs":0} 2024-03-18T01:16:46.163 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 37f51182-e4c4-11ee-95c9-87774f69a715 -- ceph osd dump --format=json 2024-03-18T01:16:46.425 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/37f51182-e4c4-11ee-95c9-87774f69a715/mon.smithi029/config 2024-03-18T01:16:46.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:16:46 smithi090 ceph-mon[36989]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-18T01:16:46.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:16:46 smithi090 ceph-mon[36989]: from='client.? 172.21.15.29:0/3027740973' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-18T01:16:46.546 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:16:46 smithi029 ceph-mon[29331]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-18T01:16:46.547 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:16:46 smithi029 ceph-mon[29331]: from='client.? 172.21.15.29:0/3027740973' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-18T01:16:48.189 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:48.187+0000 7f6ecdd87700 1 -- 172.21.15.29:0/3523736300 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6ec80fe7b0 msgr2=0x7f6ec8106e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:16:48.189 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:48.187+0000 7f6ecdd87700 1 --2- 172.21.15.29:0/3523736300 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6ec80fe7b0 0x7f6ec8106e80 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f6ebc004940 tx=0x7f6ebc009e30 comp rx=0 tx=0).stop 2024-03-18T01:16:48.189 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:48.187+0000 7f6ecdd87700 1 -- 172.21.15.29:0/3523736300 shutdown_connections 2024-03-18T01:16:48.189 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:48.187+0000 7f6ecdd87700 1 --2- 172.21.15.29:0/3523736300 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6ec80fe7b0 0x7f6ec8106e80 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:16:48.189 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:48.187+0000 7f6ecdd87700 1 --2- 172.21.15.29:0/3523736300 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6ec80fde10 0x7f6ec80fe1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:16:48.189 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:48.187+0000 7f6ecdd87700 1 -- 172.21.15.29:0/3523736300 >> 172.21.15.29:0/3523736300 conn(0x7f6ec80f7870 msgr2=0x7f6ec80f9c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:16:48.189 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:48.187+0000 7f6ecdd87700 1 -- 172.21.15.29:0/3523736300 shutdown_connections 2024-03-18T01:16:48.189 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:48.187+0000 7f6ecdd87700 1 -- 172.21.15.29:0/3523736300 wait complete. 2024-03-18T01:16:48.189 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:48.187+0000 7f6ecdd87700 1 Processor -- start 2024-03-18T01:16:48.190 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:48.187+0000 7f6ecdd87700 1 -- start start 2024-03-18T01:16:48.190 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:48.188+0000 7f6ecdd87700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6ec80fde10 0x7f6ec80fbcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:16:48.190 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:48.188+0000 7f6ecdd87700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6ec80fe7b0 0x7f6ec80fc210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:16:48.190 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:48.188+0000 7f6ecdd87700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ec80fdc20 con 0x7f6ec80fe7b0 2024-03-18T01:16:48.190 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:48.188+0000 7f6ecdd87700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ec80fc750 con 0x7f6ec80fde10 2024-03-18T01:16:48.190 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:48.188+0000 7f6ec6ffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6ec80fe7b0 0x7f6ec80fc210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:16:48.190 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:48.188+0000 7f6ec6ffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6ec80fe7b0 0x7f6ec80fc210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:38170/0 (socket says 172.21.15.29:38170) 2024-03-18T01:16:48.190 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:48.188+0000 7f6ec77fe700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6ec80fde10 0x7f6ec80fbcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:16:48.190 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:48.188+0000 7f6ec6ffd700 1 -- 172.21.15.29:0/4252394977 learned_addr learned my addr 172.21.15.29:0/4252394977 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-18T01:16:48.190 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:48.188+0000 7f6ec6ffd700 1 -- 172.21.15.29:0/4252394977 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6ec80fde10 msgr2=0x7f6ec80fbcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:16:48.190 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:48.188+0000 7f6ec6ffd700 1 --2- 172.21.15.29:0/4252394977 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6ec80fde10 0x7f6ec80fbcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:16:48.190 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:48.188+0000 7f6ec6ffd700 1 -- 172.21.15.29:0/4252394977 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ebc005040 con 0x7f6ec80fe7b0 2024-03-18T01:16:48.190 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:48.188+0000 7f6ec6ffd700 1 --2- 172.21.15.29:0/4252394977 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6ec80fe7b0 0x7f6ec80fc210 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f6ebc009380 tx=0x7f6ebc009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:16:48.190 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:48.188+0000 7f6eaffff700 1 -- 172.21.15.29:0/4252394977 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6ebc00e040 con 0x7f6ec80fe7b0 2024-03-18T01:16:48.190 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:48.188+0000 7f6eaffff700 1 -- 172.21.15.29:0/4252394977 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6ebc009af0 con 0x7f6ec80fe7b0 2024-03-18T01:16:48.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:48.188+0000 7f6ecdd87700 1 -- 172.21.15.29:0/4252394977 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ec80fc9d0 con 0x7f6ec80fe7b0 2024-03-18T01:16:48.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:48.189+0000 7f6eaffff700 1 -- 172.21.15.29:0/4252394977 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6ebc01c3f0 con 0x7f6ec80fe7b0 2024-03-18T01:16:48.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:48.189+0000 7f6ecdd87700 1 -- 172.21.15.29:0/4252394977 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ec811a0a0 con 0x7f6ec80fe7b0 2024-03-18T01:16:48.192 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:48.189+0000 7f6ecdd87700 1 -- 172.21.15.29:0/4252394977 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ec8066ad0 con 0x7f6ec80fe7b0 2024-03-18T01:16:48.192 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:48.190+0000 7f6eaffff700 1 -- 172.21.15.29:0/4252394977 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f6ebc013420 con 0x7f6ec80fe7b0 2024-03-18T01:16:48.192 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:48.190+0000 7f6eaffff700 1 --2- 172.21.15.29:0/4252394977 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f6eb00692a0 0x7f6eb006b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:16:48.192 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:48.190+0000 7f6eaffff700 1 -- 172.21.15.29:0/4252394977 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f6ebc0881a0 con 0x7f6ec80fe7b0 2024-03-18T01:16:48.192 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:48.190+0000 7f6ec77fe700 1 --2- 172.21.15.29:0/4252394977 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f6eb00692a0 0x7f6eb006b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:16:48.192 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:48.191+0000 7f6ec77fe700 1 --2- 172.21.15.29:0/4252394977 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f6eb00692a0 0x7f6eb006b750 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f6eb8004f40 tx=0x7f6eb800b040 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:16:48.195 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:48.195+0000 7f6eaffff700 1 -- 172.21.15.29:0/4252394977 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175520 (secure 0 0 0) 0x7f6ebc054790 con 0x7f6ec80fe7b0 2024-03-18T01:16:48.195 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:16:48 smithi029 ceph-mon[29331]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-18T01:16:48.390 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:48.390+0000 7f6ecdd87700 1 -- 172.21.15.29:0/4252394977 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6ec8109ea0 con 0x7f6ec80fe7b0 2024-03-18T01:16:48.391 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:48.391+0000 7f6eaffff700 1 -- 172.21.15.29:0/4252394977 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v40) v1 ==== 74+0+11035 (secure 0 0 0) 0x7f6ebc057db0 con 0x7f6ec80fe7b0 2024-03-18T01:16:48.391 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-18T01:16:48.391 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":40,"fsid":"37f51182-e4c4-11ee-95c9-87774f69a715","created":"2024-03-18T01:10:49.947064+0000","modified":"2024-03-18T01:16:42.545974+0000","last_up_change":"2024-03-18T01:16:41.540245+0000","last_in_change":"2024-03-18T01:16:23.078422+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T01:15:31.263727+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"decb820d-5c65-4317-b3f0-46522f1d1326","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":895619893},{"type":"v1","addr":"172.21.15.29:6803","nonce":895619893}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":895619893},{"type":"v1","addr":"172.21.15.29:6805","nonce":895619893}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":895619893},{"type":"v1","addr":"172.21.15.29:6809","nonce":895619893}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":895619893},{"type":"v1","addr":"172.21.15.29:6807","nonce":895619893}]},"public_addr":"172.21.15.29:6803/895619893","cluster_addr":"172.21.15.29:6805/895619893","heartbeat_back_addr":"172.21.15.29:6809/895619893","heartbeat_front_addr":"172.21.15.29:6807/895619893","state":["exists","up"]},{"osd":1,"uuid":"3b1f0295-4dd6-4bb1-9847-93bd98677db6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":2615748531},{"type":"v1","addr":"172.21.15.29:6811","nonce":2615748531}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":2615748531},{"type":"v1","addr":"172.21.15.29:6813","nonce":2615748531}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":2615748531},{"type":"v1","addr":"172.21.15.29:6817","nonce":2615748531}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":2615748531},{"type":"v1","addr":"172.21.15.29:6815","nonce":2615748531}]},"public_addr":"172.21.15.29:6811/2615748531","cluster_addr":"172.21.15.29:6813/2615748531","heartbeat_back_addr":"172.21.15.29:6817/2615748531","heartbeat_front_addr":"172.21.15.29:6815/2615748531","state":["exists","up"]},{"osd":2,"uuid":"826d30cd-7b7d-4f93-9567-c6bf6120dfe0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":576060095},{"type":"v1","addr":"172.21.15.29:6819","nonce":576060095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6820","nonce":576060095},{"type":"v1","addr":"172.21.15.29:6821","nonce":576060095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":576060095},{"type":"v1","addr":"172.21.15.29:6825","nonce":576060095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":576060095},{"type":"v1","addr":"172.21.15.29:6823","nonce":576060095}]},"public_addr":"172.21.15.29:6819/576060095","cluster_addr":"172.21.15.29:6821/576060095","heartbeat_back_addr":"172.21.15.29:6825/576060095","heartbeat_front_addr":"172.21.15.29:6823/576060095","state":["exists","up"]},{"osd":3,"uuid":"cba371da-b198-4b14-a3ac-fcf90d1a91e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":1797049062},{"type":"v1","addr":"172.21.15.90:6801","nonce":1797049062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":1797049062},{"type":"v1","addr":"172.21.15.90:6803","nonce":1797049062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":1797049062},{"type":"v1","addr":"172.21.15.90:6807","nonce":1797049062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":1797049062},{"type":"v1","addr":"172.21.15.90:6805","nonce":1797049062}]},"public_addr":"172.21.15.90:6801/1797049062","cluster_addr":"172.21.15.90:6803/1797049062","heartbeat_back_addr":"172.21.15.90:6807/1797049062","heartbeat_front_addr":"172.21.15.90:6805/1797049062","state":["exists","up"]},{"osd":4,"uuid":"c18771da-2efb-4897-a07a-ce48c94e2dee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":2675255441},{"type":"v1","addr":"172.21.15.90:6809","nonce":2675255441}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":2675255441},{"type":"v1","addr":"172.21.15.90:6811","nonce":2675255441}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":2675255441},{"type":"v1","addr":"172.21.15.90:6815","nonce":2675255441}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":2675255441},{"type":"v1","addr":"172.21.15.90:6813","nonce":2675255441}]},"public_addr":"172.21.15.90:6809/2675255441","cluster_addr":"172.21.15.90:6811/2675255441","heartbeat_back_addr":"172.21.15.90:6815/2675255441","heartbeat_front_addr":"172.21.15.90:6813/2675255441","state":["exists","up"]},{"osd":5,"uuid":"82248e69-74f9-4f46-a3fd-9224dcc8eae6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":2630668390},{"type":"v1","addr":"172.21.15.90:6817","nonce":2630668390}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6818","nonce":2630668390},{"type":"v1","addr":"172.21.15.90:6819","nonce":2630668390}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":2630668390},{"type":"v1","addr":"172.21.15.90:6823","nonce":2630668390}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6820","nonce":2630668390},{"type":"v1","addr":"172.21.15.90:6821","nonce":2630668390}]},"public_addr":"172.21.15.90:6817/2630668390","cluster_addr":"172.21.15.90:6819/2630668390","heartbeat_back_addr":"172.21.15.90:6823/2630668390","heartbeat_front_addr":"172.21.15.90:6821/2630668390","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:14:30.758603+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:14:59.452381+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:15:28.377239+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:15:51.151100+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:16:15.663658+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:16:40.030041+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.29:0/2762171838":"2024-03-19T01:13:10.926615+0000","172.21.15.29:0/3466320979":"2024-03-19T01:13:10.926615+0000","172.21.15.29:6801/2687730237":"2024-03-19T01:13:10.926615+0000","172.21.15.29:0/3723093724":"2024-03-19T01:11:59.169888+0000","172.21.15.29:6801/950856841":"2024-03-19T01:11:59.169888+0000","172.21.15.29:0/3604736621":"2024-03-19T01:11:59.169888+0000","172.21.15.29:6800/2687730237":"2024-03-19T01:13:10.926615+0000","172.21.15.29:0/3191846431":"2024-03-19T01:13:10.926615+0000","172.21.15.29:6800/1702596206":"2024-03-19T01:11:26.433683+0000","172.21.15.29:0/4093790008":"2024-03-19T01:11:26.433683+0000","172.21.15.29:6801/1702596206":"2024-03-19T01:11:26.433683+0000","172.21.15.29:0/1016951963":"2024-03-19T01:11:26.433683+0000","172.21.15.29:6800/950856841":"2024-03-19T01:11:59.169888+0000","172.21.15.29:0/2614521832":"2024-03-19T01:11:26.433683+0000","172.21.15.29:0/3826687028":"2024-03-19T01:11:59.169888+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T01:16:48.393 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:48.393+0000 7f6ecdd87700 1 -- 172.21.15.29:0/4252394977 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f6eb00692a0 msgr2=0x7f6eb006b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:16:48.393 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:48.393+0000 7f6ecdd87700 1 --2- 172.21.15.29:0/4252394977 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f6eb00692a0 0x7f6eb006b750 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f6eb8004f40 tx=0x7f6eb800b040 comp rx=0 tx=0).stop 2024-03-18T01:16:48.393 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:48.393+0000 7f6ecdd87700 1 -- 172.21.15.29:0/4252394977 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6ec80fe7b0 msgr2=0x7f6ec80fc210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:16:48.393 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:48.393+0000 7f6ecdd87700 1 --2- 172.21.15.29:0/4252394977 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6ec80fe7b0 0x7f6ec80fc210 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f6ebc009380 tx=0x7f6ebc009e30 comp rx=0 tx=0).stop 2024-03-18T01:16:48.393 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:48.393+0000 7f6ecdd87700 1 -- 172.21.15.29:0/4252394977 shutdown_connections 2024-03-18T01:16:48.394 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:48.393+0000 7f6ecdd87700 1 --2- 172.21.15.29:0/4252394977 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f6eb00692a0 0x7f6eb006b750 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:16:48.394 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:48.393+0000 7f6ecdd87700 1 --2- 172.21.15.29:0/4252394977 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6ec80fde10 0x7f6ec80fbcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:16:48.394 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:48.393+0000 7f6ecdd87700 1 --2- 172.21.15.29:0/4252394977 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6ec80fe7b0 0x7f6ec80fc210 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:16:48.394 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:48.393+0000 7f6ecdd87700 1 -- 172.21.15.29:0/4252394977 >> 172.21.15.29:0/4252394977 conn(0x7f6ec80f7870 msgr2=0x7f6ec80f8d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:16:48.394 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:48.393+0000 7f6ecdd87700 1 -- 172.21.15.29:0/4252394977 shutdown_connections 2024-03-18T01:16:48.394 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:48.393+0000 7f6ecdd87700 1 -- 172.21.15.29:0/4252394977 wait complete. 2024-03-18T01:16:48.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:16:48 smithi090 ceph-mon[36989]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-18T01:16:49.116 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:16:49 smithi029 ceph-mon[29331]: from='client.? 172.21.15.29:0/4252394977' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T01:16:49.242 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-18T01:15:31.263727+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2024-03-18T01:16:49.242 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 37f51182-e4c4-11ee-95c9-87774f69a715 -- ceph osd pool get .mgr pg_num 2024-03-18T01:16:49.420 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:16:49 smithi090 ceph-mon[36989]: from='client.? 172.21.15.29:0/4252394977' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T01:16:49.506 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/37f51182-e4c4-11ee-95c9-87774f69a715/mon.smithi029/config 2024-03-18T01:16:50.319 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:16:50 smithi029 ceph-mon[29331]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-18T01:16:50.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:16:50 smithi090 ceph-mon[36989]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-18T01:16:51.313 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:51.311+0000 7f2fec25f700 1 -- 172.21.15.29:0/3989905742 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2fe4102e00 msgr2=0x7f2fe41031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:16:51.313 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:51.311+0000 7f2fec25f700 1 --2- 172.21.15.29:0/3989905742 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2fe4102e00 0x7f2fe41031d0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f2fe0004940 tx=0x7f2fe0009e30 comp rx=0 tx=0).stop 2024-03-18T01:16:51.313 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:51.311+0000 7f2fec25f700 1 -- 172.21.15.29:0/3989905742 shutdown_connections 2024-03-18T01:16:51.313 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:51.311+0000 7f2fec25f700 1 --2- 172.21.15.29:0/3989905742 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2fe41037a0 0x7f2fe40fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:16:51.313 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:51.311+0000 7f2fec25f700 1 --2- 172.21.15.29:0/3989905742 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2fe4102e00 0x7f2fe41031d0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:16:51.313 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:51.311+0000 7f2fec25f700 1 -- 172.21.15.29:0/3989905742 >> 172.21.15.29:0/3989905742 conn(0x7f2fe40f89b0 msgr2=0x7f2fe40fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:16:51.313 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:51.311+0000 7f2fec25f700 1 -- 172.21.15.29:0/3989905742 shutdown_connections 2024-03-18T01:16:51.313 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:51.311+0000 7f2fec25f700 1 -- 172.21.15.29:0/3989905742 wait complete. 2024-03-18T01:16:51.313 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:51.311+0000 7f2fec25f700 1 Processor -- start 2024-03-18T01:16:51.313 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:51.311+0000 7f2fec25f700 1 -- start start 2024-03-18T01:16:51.313 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:51.311+0000 7f2fec25f700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2fe4102e00 0x7f2fe40702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:16:51.314 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:51.311+0000 7f2fec25f700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2fe41037a0 0x7f2fe4070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:16:51.314 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:51.311+0000 7f2fec25f700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fe40721c0 con 0x7f2fe4102e00 2024-03-18T01:16:51.314 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:51.311+0000 7f2fec25f700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fe4070d40 con 0x7f2fe41037a0 2024-03-18T01:16:51.314 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:51.312+0000 7f2fe9ffb700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2fe4102e00 0x7f2fe40702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:16:51.314 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:51.312+0000 7f2fe9ffb700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2fe4102e00 0x7f2fe40702c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:38178/0 (socket says 172.21.15.29:38178) 2024-03-18T01:16:51.314 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:51.312+0000 7f2fe9ffb700 1 -- 172.21.15.29:0/377990193 learned_addr learned my addr 172.21.15.29:0/377990193 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-18T01:16:51.314 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:51.312+0000 7f2fe97fa700 1 --2- 172.21.15.29:0/377990193 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2fe41037a0 0x7f2fe4070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:16:51.314 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:51.312+0000 7f2fe9ffb700 1 -- 172.21.15.29:0/377990193 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2fe41037a0 msgr2=0x7f2fe4070800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:16:51.314 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:51.312+0000 7f2fe9ffb700 1 --2- 172.21.15.29:0/377990193 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2fe41037a0 0x7f2fe4070800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:16:51.314 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:51.312+0000 7f2fe9ffb700 1 -- 172.21.15.29:0/377990193 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fe0005040 con 0x7f2fe4102e00 2024-03-18T01:16:51.314 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:51.312+0000 7f2fe97fa700 1 --2- 172.21.15.29:0/377990193 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2fe41037a0 0x7f2fe4070800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T01:16:51.315 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:51.312+0000 7f2fe9ffb700 1 --2- 172.21.15.29:0/377990193 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2fe4102e00 0x7f2fe40702c0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f2fe000b3b0 tx=0x7f2fe0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:16:51.315 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:51.312+0000 7f2fda7fc700 1 -- 172.21.15.29:0/377990193 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2fe000e040 con 0x7f2fe4102e00 2024-03-18T01:16:51.315 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:51.312+0000 7f2fda7fc700 1 -- 172.21.15.29:0/377990193 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2fe000bbc0 con 0x7f2fe4102e00 2024-03-18T01:16:51.315 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:51.312+0000 7f2fec25f700 1 -- 172.21.15.29:0/377990193 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2fe4070fc0 con 0x7f2fe4102e00 2024-03-18T01:16:51.315 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:51.312+0000 7f2fda7fc700 1 -- 172.21.15.29:0/377990193 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2fe001c3f0 con 0x7f2fe4102e00 2024-03-18T01:16:51.315 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:51.312+0000 7f2fec25f700 1 -- 172.21.15.29:0/377990193 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2fe411a130 con 0x7f2fe4102e00 2024-03-18T01:16:51.316 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:51.313+0000 7f2fec25f700 1 -- 172.21.15.29:0/377990193 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2fe4100110 con 0x7f2fe4102e00 2024-03-18T01:16:51.316 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:51.314+0000 7f2fda7fc700 1 -- 172.21.15.29:0/377990193 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f2fe000bd30 con 0x7f2fe4102e00 2024-03-18T01:16:51.316 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:51.314+0000 7f2fda7fc700 1 --2- 172.21.15.29:0/377990193 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f2fd00692a0 0x7f2fd006b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:16:51.316 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:51.314+0000 7f2fda7fc700 1 -- 172.21.15.29:0/377990193 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f2fe0088020 con 0x7f2fe4102e00 2024-03-18T01:16:51.316 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:51.314+0000 7f2fe97fa700 1 --2- 172.21.15.29:0/377990193 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f2fd00692a0 0x7f2fd006b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:16:51.316 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:51.315+0000 7f2fe97fa700 1 --2- 172.21.15.29:0/377990193 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f2fd00692a0 0x7f2fd006b750 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f2fd4009fd0 tx=0x7f2fd4009ee0 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:16:51.319 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:51.319+0000 7f2fda7fc700 1 -- 172.21.15.29:0/377990193 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175520 (secure 0 0 0) 0x7f2fe0054740 con 0x7f2fe4102e00 2024-03-18T01:16:51.513 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:51.512+0000 7f2fec25f700 1 -- 172.21.15.29:0/377990193 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f2fe40fe550 con 0x7f2fe4102e00 2024-03-18T01:16:51.513 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:51.512+0000 7f2fda7fc700 1 -- 172.21.15.29:0/377990193 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v40) v1 ==== 93+0+10 (secure 0 0 0) 0x7f2fe0057d60 con 0x7f2fe4102e00 2024-03-18T01:16:51.513 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 1 2024-03-18T01:16:51.515 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:51.515+0000 7f2fec25f700 1 -- 172.21.15.29:0/377990193 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f2fd00692a0 msgr2=0x7f2fd006b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:16:51.516 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:51.515+0000 7f2fec25f700 1 --2- 172.21.15.29:0/377990193 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f2fd00692a0 0x7f2fd006b750 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f2fd4009fd0 tx=0x7f2fd4009ee0 comp rx=0 tx=0).stop 2024-03-18T01:16:51.516 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:51.515+0000 7f2fec25f700 1 -- 172.21.15.29:0/377990193 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2fe4102e00 msgr2=0x7f2fe40702c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:16:51.516 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:51.515+0000 7f2fec25f700 1 --2- 172.21.15.29:0/377990193 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2fe4102e00 0x7f2fe40702c0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f2fe000b3b0 tx=0x7f2fe0009e30 comp rx=0 tx=0).stop 2024-03-18T01:16:51.516 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:51.515+0000 7f2fec25f700 1 -- 172.21.15.29:0/377990193 shutdown_connections 2024-03-18T01:16:51.516 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:51.515+0000 7f2fec25f700 1 --2- 172.21.15.29:0/377990193 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f2fd00692a0 0x7f2fd006b750 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:16:51.516 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:51.515+0000 7f2fec25f700 1 --2- 172.21.15.29:0/377990193 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2fe4102e00 0x7f2fe40702c0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:16:51.516 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:51.515+0000 7f2fec25f700 1 --2- 172.21.15.29:0/377990193 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2fe41037a0 0x7f2fe4070800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:16:51.516 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:51.515+0000 7f2fec25f700 1 -- 172.21.15.29:0/377990193 >> 172.21.15.29:0/377990193 conn(0x7f2fe40f89b0 msgr2=0x7f2fe41073a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:16:51.516 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:51.515+0000 7f2fec25f700 1 -- 172.21.15.29:0/377990193 shutdown_connections 2024-03-18T01:16:51.516 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:51.515+0000 7f2fec25f700 1 -- 172.21.15.29:0/377990193 wait complete. 2024-03-18T01:16:52.193 INFO:tasks.cephadm:Setting up client nodes... 2024-03-18T01:16:52.194 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 37f51182-e4c4-11ee-95c9-87774f69a715 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-03-18T01:16:52.220 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:16:52 smithi029 ceph-mon[29331]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-18T01:16:52.220 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:16:52 smithi029 ceph-mon[29331]: from='client.? 172.21.15.29:0/377990193' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T01:16:52.455 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/37f51182-e4c4-11ee-95c9-87774f69a715/mon.smithi029/config 2024-03-18T01:16:52.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:16:52 smithi090 ceph-mon[36989]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-18T01:16:52.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:16:52 smithi090 ceph-mon[36989]: from='client.? 172.21.15.29:0/377990193' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T01:16:54.134 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:54.132+0000 7f91566ba700 1 -- 172.21.15.29:0/851383178 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f91500fe060 msgr2=0x7f91500fe4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:16:54.134 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:54.132+0000 7f91566ba700 1 --2- 172.21.15.29:0/851383178 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f91500fe060 0x7f91500fe4d0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f9144004940 tx=0x7f9144009e30 comp rx=0 tx=0).stop 2024-03-18T01:16:54.134 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:54.132+0000 7f91566ba700 1 -- 172.21.15.29:0/851383178 shutdown_connections 2024-03-18T01:16:54.134 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:54.132+0000 7f91566ba700 1 --2- 172.21.15.29:0/851383178 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f91500fe060 0x7f91500fe4d0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:16:54.135 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:54.132+0000 7f91566ba700 1 --2- 172.21.15.29:0/851383178 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f91500fce60 0x7f91500fd270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:16:54.135 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:54.132+0000 7f91566ba700 1 -- 172.21.15.29:0/851383178 >> 172.21.15.29:0/851383178 conn(0x7f91500f8570 msgr2=0x7f91500fa9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:16:54.135 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:54.132+0000 7f91566ba700 1 -- 172.21.15.29:0/851383178 shutdown_connections 2024-03-18T01:16:54.135 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:54.132+0000 7f91566ba700 1 -- 172.21.15.29:0/851383178 wait complete. 2024-03-18T01:16:54.135 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:54.132+0000 7f91566ba700 1 Processor -- start 2024-03-18T01:16:54.135 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:54.133+0000 7f91566ba700 1 -- start start 2024-03-18T01:16:54.135 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:54.133+0000 7f91566ba700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f91500fce60 0x7f915010d920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:16:54.135 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:54.133+0000 7f91566ba700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f91500fe060 0x7f915010de60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:16:54.135 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:54.133+0000 7f91566ba700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f915010e460 con 0x7f91500fe060 2024-03-18T01:16:54.135 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:54.133+0000 7f91566ba700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f915010e5d0 con 0x7f91500fce60 2024-03-18T01:16:54.135 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:54.133+0000 7f914f7fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f91500fe060 0x7f915010de60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:16:54.135 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:54.133+0000 7f914f7fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f91500fe060 0x7f915010de60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45162/0 (socket says 172.21.15.29:45162) 2024-03-18T01:16:54.135 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:54.133+0000 7f914f7fe700 1 -- 172.21.15.29:0/1602226199 learned_addr learned my addr 172.21.15.29:0/1602226199 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-18T01:16:54.135 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:54.133+0000 7f914ffff700 1 --2- 172.21.15.29:0/1602226199 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f91500fce60 0x7f915010d920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:16:54.135 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:54.133+0000 7f914f7fe700 1 -- 172.21.15.29:0/1602226199 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f91500fce60 msgr2=0x7f915010d920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:16:54.136 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:54.133+0000 7f914f7fe700 1 --2- 172.21.15.29:0/1602226199 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f91500fce60 0x7f915010d920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:16:54.136 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:54.133+0000 7f914f7fe700 1 -- 172.21.15.29:0/1602226199 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9144005040 con 0x7f91500fe060 2024-03-18T01:16:54.136 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:54.133+0000 7f914f7fe700 1 --2- 172.21.15.29:0/1602226199 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f91500fe060 0x7f915010de60 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f914400b3b0 tx=0x7f9144009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:16:54.136 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:54.133+0000 7f914cff9700 1 -- 172.21.15.29:0/1602226199 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f914400e040 con 0x7f91500fe060 2024-03-18T01:16:54.136 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:54.134+0000 7f914cff9700 1 -- 172.21.15.29:0/1602226199 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f914400bbf0 con 0x7f91500fe060 2024-03-18T01:16:54.136 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:54.134+0000 7f91566ba700 1 -- 172.21.15.29:0/1602226199 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9150112f60 con 0x7f91500fe060 2024-03-18T01:16:54.136 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:54.134+0000 7f914cff9700 1 -- 172.21.15.29:0/1602226199 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f914401c630 con 0x7f91500fe060 2024-03-18T01:16:54.136 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:54.134+0000 7f91566ba700 1 -- 172.21.15.29:0/1602226199 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91501134f0 con 0x7f91500fe060 2024-03-18T01:16:54.137 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:54.135+0000 7f91566ba700 1 -- 172.21.15.29:0/1602226199 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f915004c710 con 0x7f91500fe060 2024-03-18T01:16:54.137 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:54.135+0000 7f914cff9700 1 -- 172.21.15.29:0/1602226199 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f914400bd60 con 0x7f91500fe060 2024-03-18T01:16:54.137 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:54.136+0000 7f914cff9700 1 --2- 172.21.15.29:0/1602226199 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f91380692a0 0x7f913806b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:16:54.137 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:54.136+0000 7f914ffff700 1 --2- 172.21.15.29:0/1602226199 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f91380692a0 0x7f913806b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:16:54.137 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:54.136+0000 7f914cff9700 1 -- 172.21.15.29:0/1602226199 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f9144087b30 con 0x7f91500fe060 2024-03-18T01:16:54.137 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:54.136+0000 7f914ffff700 1 --2- 172.21.15.29:0/1602226199 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f91380692a0 0x7f913806b750 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f91500fdbd0 tx=0x7f914000b040 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:16:54.140 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:54.140+0000 7f914cff9700 1 -- 172.21.15.29:0/1602226199 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175520 (secure 0 0 0) 0x7f9144057c30 con 0x7f91500fe060 2024-03-18T01:16:54.319 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:16:54 smithi029 ceph-mon[29331]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-18T01:16:54.409 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:54.409+0000 7f91566ba700 1 -- 172.21.15.29:0/1602226199 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) v1 -- 0x7f9150113680 con 0x7f91500fe060 2024-03-18T01:16:54.413 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:54.413+0000 7f914cff9700 1 -- 172.21.15.29:0/1602226199 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v14) v1 ==== 170+0+59 (secure 0 0 0) 0x7f91440577c0 con 0x7f91500fe060 2024-03-18T01:16:54.413 INFO:teuthology.orchestra.run.smithi029.stdout:[client.0] 2024-03-18T01:16:54.414 INFO:teuthology.orchestra.run.smithi029.stdout: key = AQAGlvdl/np9GBAAFVlJO2+iOlSTpWbYDu/oDg== 2024-03-18T01:16:54.416 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:54.415+0000 7f91566ba700 1 -- 172.21.15.29:0/1602226199 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f91380692a0 msgr2=0x7f913806b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:16:54.416 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:54.415+0000 7f91566ba700 1 --2- 172.21.15.29:0/1602226199 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f91380692a0 0x7f913806b750 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f91500fdbd0 tx=0x7f914000b040 comp rx=0 tx=0).stop 2024-03-18T01:16:54.416 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:54.415+0000 7f91566ba700 1 -- 172.21.15.29:0/1602226199 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f91500fe060 msgr2=0x7f915010de60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:16:54.416 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:54.415+0000 7f91566ba700 1 --2- 172.21.15.29:0/1602226199 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f91500fe060 0x7f915010de60 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f914400b3b0 tx=0x7f9144009e30 comp rx=0 tx=0).stop 2024-03-18T01:16:54.416 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:54.415+0000 7f91566ba700 1 -- 172.21.15.29:0/1602226199 shutdown_connections 2024-03-18T01:16:54.416 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:54.415+0000 7f91566ba700 1 --2- 172.21.15.29:0/1602226199 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f91380692a0 0x7f913806b750 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:16:54.416 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:54.415+0000 7f91566ba700 1 --2- 172.21.15.29:0/1602226199 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f91500fce60 0x7f915010d920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:16:54.416 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:54.415+0000 7f91566ba700 1 --2- 172.21.15.29:0/1602226199 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f91500fe060 0x7f915010de60 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:16:54.416 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:54.415+0000 7f91566ba700 1 -- 172.21.15.29:0/1602226199 >> 172.21.15.29:0/1602226199 conn(0x7f91500f8570 msgr2=0x7f91500f9c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:16:54.416 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:54.415+0000 7f91566ba700 1 -- 172.21.15.29:0/1602226199 shutdown_connections 2024-03-18T01:16:54.416 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:54.415+0000 7f91566ba700 1 -- 172.21.15.29:0/1602226199 wait complete. 2024-03-18T01:16:54.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:16:54 smithi090 ceph-mon[36989]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-18T01:16:55.029 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-03-18T01:16:55.029 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-03-18T01:16:55.029 DEBUG:teuthology.orchestra.run.smithi029:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-18T01:16:55.113 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 37f51182-e4c4-11ee-95c9-87774f69a715 -- ceph auth get-or-create client.1 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-03-18T01:16:55.318 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:16:55 smithi029 ceph-mon[29331]: from='client.? 172.21.15.29:0/1602226199' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-18T01:16:55.319 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:16:55 smithi029 ceph-mon[29331]: from='client.? 172.21.15.29:0/1602226199' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-18T01:16:55.384 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/37f51182-e4c4-11ee-95c9-87774f69a715/mon.smithi090/config 2024-03-18T01:16:55.507 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:16:55 smithi090 ceph-mon[36989]: from='client.? 172.21.15.29:0/1602226199' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-18T01:16:55.507 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:16:55 smithi090 ceph-mon[36989]: from='client.? 172.21.15.29:0/1602226199' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-18T01:16:56.197 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:16:56 smithi090 ceph-mon[36989]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-18T01:16:56.197 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:16:56 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-18T01:16:56.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:16:56 smithi029 ceph-mon[29331]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-18T01:16:56.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:16:56 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-18T01:16:57.115 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:16:56.991+0000 7f1ec6394700 1 -- 172.21.15.90:0/630605819 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1ec00fcf90 msgr2=0x7f1ec00fd400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:16:57.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:16:56.991+0000 7f1ec6394700 1 --2- 172.21.15.90:0/630605819 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1ec00fcf90 0x7f1ec00fd400 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f1eb4004940 tx=0x7f1eb4009e30 comp rx=0 tx=0).stop 2024-03-18T01:16:57.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:16:56.992+0000 7f1ec6394700 1 -- 172.21.15.90:0/630605819 shutdown_connections 2024-03-18T01:16:57.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:16:56.992+0000 7f1ec6394700 1 --2- 172.21.15.90:0/630605819 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1ec00fcf90 0x7f1ec00fd400 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:16:57.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:16:56.992+0000 7f1ec6394700 1 --2- 172.21.15.90:0/630605819 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1ec00fbce0 0x7f1ec00fc0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:16:57.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:16:56.992+0000 7f1ec6394700 1 -- 172.21.15.90:0/630605819 >> 172.21.15.90:0/630605819 conn(0x7f1ec00f7480 msgr2=0x7f1ec00f9890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:16:57.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:16:56.992+0000 7f1ec6394700 1 -- 172.21.15.90:0/630605819 shutdown_connections 2024-03-18T01:16:57.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:16:56.992+0000 7f1ec6394700 1 -- 172.21.15.90:0/630605819 wait complete. 2024-03-18T01:16:57.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:16:56.992+0000 7f1ec6394700 1 Processor -- start 2024-03-18T01:16:57.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:16:56.992+0000 7f1ec6394700 1 -- start start 2024-03-18T01:16:57.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:16:56.992+0000 7f1ec6394700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1ec00fbce0 0x7f1ec0066d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:16:57.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:16:56.992+0000 7f1ec6394700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1ec00fcf90 0x7f1ec0112730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:16:57.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:16:56.992+0000 7f1ec6394700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ec0112c70 con 0x7f1ec00fbce0 2024-03-18T01:16:57.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:16:56.992+0000 7f1ec6394700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ec0112de0 con 0x7f1ec00fcf90 2024-03-18T01:16:57.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:16:56.993+0000 7f1ebf7fe700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1ec00fcf90 0x7f1ec0112730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:16:57.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:16:56.993+0000 7f1ebf7fe700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1ec00fcf90 0x7f1ec0112730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:43072/0 (socket says 172.21.15.90:43072) 2024-03-18T01:16:57.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:16:56.993+0000 7f1ebf7fe700 1 -- 172.21.15.90:0/618373255 learned_addr learned my addr 172.21.15.90:0/618373255 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-18T01:16:57.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:16:56.993+0000 7f1ebffff700 1 --2- 172.21.15.90:0/618373255 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1ec00fbce0 0x7f1ec0066d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:16:57.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:16:56.993+0000 7f1ebf7fe700 1 -- 172.21.15.90:0/618373255 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1ec00fbce0 msgr2=0x7f1ec0066d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:16:57.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:16:56.993+0000 7f1ebf7fe700 1 --2- 172.21.15.90:0/618373255 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1ec00fbce0 0x7f1ec0066d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:16:57.225 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:16:56.993+0000 7f1ebf7fe700 1 -- 172.21.15.90:0/618373255 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1eb4005040 con 0x7f1ec00fcf90 2024-03-18T01:16:57.225 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:16:56.993+0000 7f1ebf7fe700 1 --2- 172.21.15.90:0/618373255 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1ec00fcf90 0x7f1ec0112730 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f1eb4009150 tx=0x7f1eb4009e30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:16:57.225 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:16:56.993+0000 7f1ebcff9700 1 -- 172.21.15.90:0/618373255 <== mon.1 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1eb400e040 con 0x7f1ec00fcf90 2024-03-18T01:16:57.225 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:16:56.993+0000 7f1ec6394700 1 -- 172.21.15.90:0/618373255 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ec0113060 con 0x7f1ec00fcf90 2024-03-18T01:16:57.225 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:16:56.993+0000 7f1ebcff9700 1 -- 172.21.15.90:0/618373255 <== mon.1 v2:172.21.15.90:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1eb400bbf0 con 0x7f1ec00fcf90 2024-03-18T01:16:57.225 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:16:56.993+0000 7f1ebcff9700 1 -- 172.21.15.90:0/618373255 <== mon.1 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1eb401c630 con 0x7f1ec00fcf90 2024-03-18T01:16:57.225 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:16:56.993+0000 7f1ec6394700 1 -- 172.21.15.90:0/618373255 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ec0113620 con 0x7f1ec00fcf90 2024-03-18T01:16:57.225 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:16:56.994+0000 7f1ec6394700 1 -- 172.21.15.90:0/618373255 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ec004c710 con 0x7f1ec00fcf90 2024-03-18T01:16:57.225 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:16:56.995+0000 7f1ebcff9700 1 -- 172.21.15.90:0/618373255 <== mon.1 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f1eb400bd60 con 0x7f1ec00fcf90 2024-03-18T01:16:57.225 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:16:56.995+0000 7f1ebcff9700 1 --2- 172.21.15.90:0/618373255 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f1ea80692b0 0x7f1ea806b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:16:57.225 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:16:56.995+0000 7f1ebcff9700 1 -- 172.21.15.90:0/618373255 <== mon.1 v2:172.21.15.90:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f1eb40881b0 con 0x7f1ec00fcf90 2024-03-18T01:16:57.225 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:16:56.995+0000 7f1ebffff700 1 --2- 172.21.15.90:0/618373255 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f1ea80692b0 0x7f1ea806b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:16:57.225 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:16:56.996+0000 7f1ebffff700 1 --2- 172.21.15.90:0/618373255 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f1ea80692b0 0x7f1ea806b760 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f1eb0009bf0 tx=0x7f1eb00091a0 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:16:57.225 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:16:56.999+0000 7f1ebcff9700 1 -- 172.21.15.90:0/618373255 <== mon.1 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175520 (secure 0 0 0) 0x7f1eb4058360 con 0x7f1ec00fcf90 2024-03-18T01:16:57.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:16:57.255+0000 7f1ec6394700 1 -- 172.21.15.90:0/618373255 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) v1 -- 0x7f1ec0104850 con 0x7f1ec00fcf90 2024-03-18T01:16:57.259 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:16:57.259+0000 7f1ebcff9700 1 -- 172.21.15.90:0/618373255 <== mon.1 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v15) v1 ==== 170+0+59 (secure 0 0 0) 0x7f1eb4013e60 con 0x7f1ec00fcf90 2024-03-18T01:16:57.260 INFO:teuthology.orchestra.run.smithi090.stdout:[client.1] 2024-03-18T01:16:57.260 INFO:teuthology.orchestra.run.smithi090.stdout: key = AQAJlvdlUgpUDxAATF0u6bjJ/5jaYSRW1F6BOQ== 2024-03-18T01:16:57.262 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:16:57.261+0000 7f1ec6394700 1 -- 172.21.15.90:0/618373255 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f1ea80692b0 msgr2=0x7f1ea806b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:16:57.262 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:16:57.261+0000 7f1ec6394700 1 --2- 172.21.15.90:0/618373255 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f1ea80692b0 0x7f1ea806b760 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f1eb0009bf0 tx=0x7f1eb00091a0 comp rx=0 tx=0).stop 2024-03-18T01:16:57.262 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:16:57.261+0000 7f1ec6394700 1 -- 172.21.15.90:0/618373255 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1ec00fcf90 msgr2=0x7f1ec0112730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:16:57.262 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:16:57.261+0000 7f1ec6394700 1 --2- 172.21.15.90:0/618373255 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1ec00fcf90 0x7f1ec0112730 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f1eb4009150 tx=0x7f1eb4009e30 comp rx=0 tx=0).stop 2024-03-18T01:16:57.262 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:16:57.261+0000 7f1ec6394700 1 -- 172.21.15.90:0/618373255 shutdown_connections 2024-03-18T01:16:57.262 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:16:57.261+0000 7f1ec6394700 1 --2- 172.21.15.90:0/618373255 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f1ea80692b0 0x7f1ea806b760 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:16:57.262 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:16:57.261+0000 7f1ec6394700 1 --2- 172.21.15.90:0/618373255 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1ec00fbce0 0x7f1ec0066d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:16:57.262 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:16:57.261+0000 7f1ec6394700 1 --2- 172.21.15.90:0/618373255 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1ec00fcf90 0x7f1ec0112730 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:16:57.262 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:16:57.261+0000 7f1ec6394700 1 -- 172.21.15.90:0/618373255 >> 172.21.15.90:0/618373255 conn(0x7f1ec00f7480 msgr2=0x7f1ec00f8940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:16:57.262 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:16:57.261+0000 7f1ec6394700 1 -- 172.21.15.90:0/618373255 shutdown_connections 2024-03-18T01:16:57.262 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:16:57.261+0000 7f1ec6394700 1 -- 172.21.15.90:0/618373255 wait complete. 2024-03-18T01:16:57.842 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-03-18T01:16:57.842 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd of=/etc/ceph/ceph.client.1.keyring 2024-03-18T01:16:57.842 DEBUG:teuthology.orchestra.run.smithi090:> sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-03-18T01:16:57.885 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-03-18T01:16:57.886 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-03-18T01:16:57.886 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 37f51182-e4c4-11ee-95c9-87774f69a715 -- ceph mgr dump --format=json 2024-03-18T01:16:58.158 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/37f51182-e4c4-11ee-95c9-87774f69a715/mon.smithi029/config 2024-03-18T01:16:58.280 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:16:58 smithi029 ceph-mon[29331]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-18T01:16:58.280 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:16:58 smithi029 ceph-mon[29331]: from='client.? 172.21.15.90:0/618373255' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-18T01:16:58.280 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:16:58 smithi029 ceph-mon[29331]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-18T01:16:58.280 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:16:58 smithi029 ceph-mon[29331]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-18T01:16:58.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:16:58 smithi090 ceph-mon[36989]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-18T01:16:58.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:16:58 smithi090 ceph-mon[36989]: from='client.? 172.21.15.90:0/618373255' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-18T01:16:58.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:16:58 smithi090 ceph-mon[36989]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-18T01:16:58.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:16:58 smithi090 ceph-mon[36989]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-18T01:16:59.811 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:59.804+0000 7fae24206700 1 -- 172.21.15.29:0/582180112 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fae1c0fe7f0 msgr2=0x7fae1c106ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:16:59.811 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:59.804+0000 7fae24206700 1 --2- 172.21.15.29:0/582180112 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fae1c0fe7f0 0x7fae1c106ec0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fae18004940 tx=0x7fae18009e30 comp rx=0 tx=0).stop 2024-03-18T01:16:59.811 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:59.804+0000 7fae24206700 1 -- 172.21.15.29:0/582180112 shutdown_connections 2024-03-18T01:16:59.811 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:59.804+0000 7fae24206700 1 --2- 172.21.15.29:0/582180112 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fae1c0fe7f0 0x7fae1c106ec0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:16:59.812 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:59.804+0000 7fae24206700 1 --2- 172.21.15.29:0/582180112 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fae1c0fde50 0x7fae1c0fe220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:16:59.812 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:59.804+0000 7fae24206700 1 -- 172.21.15.29:0/582180112 >> 172.21.15.29:0/582180112 conn(0x7fae1c0f7870 msgr2=0x7fae1c0f9c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:16:59.812 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:59.804+0000 7fae24206700 1 -- 172.21.15.29:0/582180112 shutdown_connections 2024-03-18T01:16:59.812 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:59.804+0000 7fae24206700 1 -- 172.21.15.29:0/582180112 wait complete. 2024-03-18T01:16:59.812 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:59.805+0000 7fae24206700 1 Processor -- start 2024-03-18T01:16:59.812 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:59.805+0000 7fae24206700 1 -- start start 2024-03-18T01:16:59.812 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:59.805+0000 7fae24206700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fae1c0fde50 0x7fae1c1139f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:16:59.812 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:59.805+0000 7fae24206700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fae1c0fe7f0 0x7fae1c113f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:16:59.812 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:59.805+0000 7fae24206700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae1c114580 con 0x7fae1c0fe7f0 2024-03-18T01:16:59.812 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:59.805+0000 7fae24206700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae1c10db80 con 0x7fae1c0fde50 2024-03-18T01:16:59.812 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:59.805+0000 7fae217a1700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fae1c0fe7f0 0x7fae1c113f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:16:59.812 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:59.805+0000 7fae217a1700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fae1c0fe7f0 0x7fae1c113f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45184/0 (socket says 172.21.15.29:45184) 2024-03-18T01:16:59.812 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:59.805+0000 7fae217a1700 1 -- 172.21.15.29:0/3418986996 learned_addr learned my addr 172.21.15.29:0/3418986996 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-18T01:16:59.812 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:59.805+0000 7fae21fa2700 1 --2- 172.21.15.29:0/3418986996 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fae1c0fde50 0x7fae1c1139f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:16:59.812 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:59.805+0000 7fae217a1700 1 -- 172.21.15.29:0/3418986996 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fae1c0fde50 msgr2=0x7fae1c1139f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:16:59.812 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:59.805+0000 7fae217a1700 1 --2- 172.21.15.29:0/3418986996 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fae1c0fde50 0x7fae1c1139f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:16:59.813 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:59.805+0000 7fae217a1700 1 -- 172.21.15.29:0/3418986996 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae18005040 con 0x7fae1c0fe7f0 2024-03-18T01:16:59.813 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:59.806+0000 7fae217a1700 1 --2- 172.21.15.29:0/3418986996 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fae1c0fe7f0 0x7fae1c113f30 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fae1800b3b0 tx=0x7fae18009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:16:59.813 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:59.806+0000 7fae127fc700 1 -- 172.21.15.29:0/3418986996 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fae1800e040 con 0x7fae1c0fe7f0 2024-03-18T01:16:59.813 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:59.806+0000 7fae127fc700 1 -- 172.21.15.29:0/3418986996 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fae18013420 con 0x7fae1c0fe7f0 2024-03-18T01:16:59.813 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:59.806+0000 7fae24206700 1 -- 172.21.15.29:0/3418986996 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae1c10de00 con 0x7fae1c0fe7f0 2024-03-18T01:16:59.813 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:59.806+0000 7fae127fc700 1 -- 172.21.15.29:0/3418986996 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fae1801c5f0 con 0x7fae1c0fe7f0 2024-03-18T01:16:59.813 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:59.806+0000 7fae24206700 1 -- 172.21.15.29:0/3418986996 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae1c10e340 con 0x7fae1c0fe7f0 2024-03-18T01:16:59.813 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:59.807+0000 7fae24206700 1 -- 172.21.15.29:0/3418986996 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae1c101980 con 0x7fae1c0fe7f0 2024-03-18T01:16:59.813 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:59.807+0000 7fae127fc700 1 -- 172.21.15.29:0/3418986996 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7fae18013590 con 0x7fae1c0fe7f0 2024-03-18T01:16:59.813 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:59.808+0000 7fae127fc700 1 --2- 172.21.15.29:0/3418986996 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7fae080692a0 0x7fae0806b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:16:59.813 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:59.808+0000 7fae127fc700 1 -- 172.21.15.29:0/3418986996 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fae18088020 con 0x7fae1c0fe7f0 2024-03-18T01:16:59.813 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:59.808+0000 7fae21fa2700 1 --2- 172.21.15.29:0/3418986996 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7fae080692a0 0x7fae0806b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:16:59.813 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:59.808+0000 7fae21fa2700 1 --2- 172.21.15.29:0/3418986996 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7fae080692a0 0x7fae0806b750 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fae0c004f60 tx=0x7fae0c00b040 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:16:59.814 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:16:59.812+0000 7fae127fc700 1 -- 172.21.15.29:0/3418986996 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175520 (secure 0 0 0) 0x7fae18058120 con 0x7fae1c0fe7f0 2024-03-18T01:17:00.060 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:00.059+0000 7fae24206700 1 -- 172.21.15.29:0/3418986996 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7fae1c10efd0 con 0x7fae1c0fe7f0 2024-03-18T01:17:00.063 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:00.062+0000 7fae127fc700 1 -- 172.21.15.29:0/3418986996 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v19) v1 ==== 74+0+167169 (secure 0 0 0) 0x7fae18057cb0 con 0x7fae1c0fe7f0 2024-03-18T01:17:00.066 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-18T01:17:00.067 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:00.064+0000 7fae24206700 1 -- 172.21.15.29:0/3418986996 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7fae080692a0 msgr2=0x7fae0806b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:17:00.067 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:00.064+0000 7fae24206700 1 --2- 172.21.15.29:0/3418986996 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7fae080692a0 0x7fae0806b750 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fae0c004f60 tx=0x7fae0c00b040 comp rx=0 tx=0).stop 2024-03-18T01:17:00.067 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:00.064+0000 7fae24206700 1 -- 172.21.15.29:0/3418986996 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fae1c0fe7f0 msgr2=0x7fae1c113f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:17:00.067 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:00.064+0000 7fae24206700 1 --2- 172.21.15.29:0/3418986996 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fae1c0fe7f0 0x7fae1c113f30 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fae1800b3b0 tx=0x7fae18009e30 comp rx=0 tx=0).stop 2024-03-18T01:17:00.067 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:00.064+0000 7fae24206700 1 -- 172.21.15.29:0/3418986996 shutdown_connections 2024-03-18T01:17:00.067 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:00.064+0000 7fae24206700 1 --2- 172.21.15.29:0/3418986996 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7fae080692a0 0x7fae0806b750 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:17:00.067 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:00.064+0000 7fae24206700 1 --2- 172.21.15.29:0/3418986996 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fae1c0fde50 0x7fae1c1139f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:17:00.067 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:00.064+0000 7fae24206700 1 --2- 172.21.15.29:0/3418986996 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fae1c0fe7f0 0x7fae1c113f30 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:17:00.067 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:00.064+0000 7fae24206700 1 -- 172.21.15.29:0/3418986996 >> 172.21.15.29:0/3418986996 conn(0x7fae1c0f7870 msgr2=0x7fae1c0ff3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:17:00.067 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:00.064+0000 7fae24206700 1 -- 172.21.15.29:0/3418986996 shutdown_connections 2024-03-18T01:17:00.068 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:00.064+0000 7fae24206700 1 -- 172.21.15.29:0/3418986996 wait complete. 2024-03-18T01:17:00.319 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:17:00 smithi029 ceph-mon[29331]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-18T01:17:00.319 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:17:00 smithi029 ceph-mon[29331]: from='client.? 172.21.15.29:0/3418986996' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-18T01:17:00.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:17:00 smithi090 ceph-mon[36989]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-18T01:17:00.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:17:00 smithi090 ceph-mon[36989]: from='client.? 172.21.15.29:0/3418986996' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-18T01:17:00.785 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":19,"active_gid":14208,"active_name":"smithi029.qviwhl","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":1641454714},{"type":"v1","addr":"172.21.15.29:6801","nonce":1641454714}]},"active_addr":"172.21.15.29:6801/1641454714","active_change":"2024-03-18T01:13:10.926722+0000","active_mgr_features":4540138320759226367,"available":true,"standbys":[{"gid":14230,"name":"smithi090.pnqedy","mgr_features":4540138320759226367,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.29:8443/","prometheus":"http://172.21.15.29:9283/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":5,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.29:0","nonce":2592619837}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.29:0","nonce":1087527911}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.29:0","nonce":2550980235}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.29:0","nonce":4161314673}]}]} 2024-03-18T01:17:00.788 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-03-18T01:17:00.788 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-03-18T01:17:00.788 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 37f51182-e4c4-11ee-95c9-87774f69a715 -- ceph osd dump --format=json 2024-03-18T01:17:01.053 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/37f51182-e4c4-11ee-95c9-87774f69a715/mon.smithi029/config 2024-03-18T01:17:02.466 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:17:02 smithi029 ceph-mon[29331]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-18T01:17:02.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:17:02 smithi090 ceph-mon[36989]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-18T01:17:02.808 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:02.805+0000 7fc769d9c700 1 -- 172.21.15.29:0/2851726299 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc764102710 msgr2=0x7fc7640fbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:17:02.808 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:02.805+0000 7fc769d9c700 1 --2- 172.21.15.29:0/2851726299 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc764102710 0x7fc7640fbd80 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fc758004940 tx=0x7fc758009e30 comp rx=0 tx=0).stop 2024-03-18T01:17:02.808 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:02.805+0000 7fc769d9c700 1 -- 172.21.15.29:0/2851726299 shutdown_connections 2024-03-18T01:17:02.808 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:02.805+0000 7fc769d9c700 1 --2- 172.21.15.29:0/2851726299 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc764102710 0x7fc7640fbd80 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:17:02.808 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:02.805+0000 7fc769d9c700 1 --2- 172.21.15.29:0/2851726299 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc764101d70 0x7fc764102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:17:02.809 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:02.805+0000 7fc769d9c700 1 -- 172.21.15.29:0/2851726299 >> 172.21.15.29:0/2851726299 conn(0x7fc764072bc0 msgr2=0x7fc764072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:17:02.809 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:02.805+0000 7fc769d9c700 1 -- 172.21.15.29:0/2851726299 shutdown_connections 2024-03-18T01:17:02.809 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:02.805+0000 7fc769d9c700 1 -- 172.21.15.29:0/2851726299 wait complete. 2024-03-18T01:17:02.809 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:02.806+0000 7fc769d9c700 1 Processor -- start 2024-03-18T01:17:02.809 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:02.806+0000 7fc769d9c700 1 -- start start 2024-03-18T01:17:02.809 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:02.806+0000 7fc769d9c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc764101d70 0x7fc76410ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:17:02.809 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:02.806+0000 7fc769d9c700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc764102710 0x7fc76410f220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:17:02.809 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:02.806+0000 7fc769d9c700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc76410f760 con 0x7fc764101d70 2024-03-18T01:17:02.809 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:02.806+0000 7fc769d9c700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc76410f8a0 con 0x7fc764102710 2024-03-18T01:17:02.809 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:02.806+0000 7fc7637fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc764101d70 0x7fc76410ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:17:02.809 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:02.806+0000 7fc7637fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc764101d70 0x7fc76410ece0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51190/0 (socket says 172.21.15.29:51190) 2024-03-18T01:17:02.809 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:02.806+0000 7fc7637fe700 1 -- 172.21.15.29:0/3508698012 learned_addr learned my addr 172.21.15.29:0/3508698012 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-18T01:17:02.809 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:02.806+0000 7fc762ffd700 1 --2- 172.21.15.29:0/3508698012 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc764102710 0x7fc76410f220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:17:02.809 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:02.807+0000 7fc7637fe700 1 -- 172.21.15.29:0/3508698012 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc764102710 msgr2=0x7fc76410f220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:17:02.809 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:02.807+0000 7fc7637fe700 1 --2- 172.21.15.29:0/3508698012 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc764102710 0x7fc76410f220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:17:02.810 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:02.807+0000 7fc7637fe700 1 -- 172.21.15.29:0/3508698012 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc758005040 con 0x7fc764101d70 2024-03-18T01:17:02.810 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:02.807+0000 7fc7637fe700 1 --2- 172.21.15.29:0/3508698012 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc764101d70 0x7fc76410ece0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fc754002a50 tx=0x7fc7540046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:17:02.810 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:02.807+0000 7fc74bfff700 1 -- 172.21.15.29:0/3508698012 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc754007cc0 con 0x7fc764101d70 2024-03-18T01:17:02.810 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:02.807+0000 7fc74bfff700 1 -- 172.21.15.29:0/3508698012 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc75400f040 con 0x7fc764101d70 2024-03-18T01:17:02.810 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:02.807+0000 7fc769d9c700 1 -- 172.21.15.29:0/3508698012 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc764117b90 con 0x7fc764101d70 2024-03-18T01:17:02.810 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:02.807+0000 7fc74bfff700 1 -- 172.21.15.29:0/3508698012 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc75400da30 con 0x7fc764101d70 2024-03-18T01:17:02.810 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:02.807+0000 7fc769d9c700 1 -- 172.21.15.29:0/3508698012 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc764118120 con 0x7fc764101d70 2024-03-18T01:17:02.811 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:02.808+0000 7fc769d9c700 1 -- 172.21.15.29:0/3508698012 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc764104240 con 0x7fc764101d70 2024-03-18T01:17:02.811 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:02.809+0000 7fc74bfff700 1 -- 172.21.15.29:0/3508698012 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7fc75400dbf0 con 0x7fc764101d70 2024-03-18T01:17:02.811 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:02.809+0000 7fc74bfff700 1 --2- 172.21.15.29:0/3508698012 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7fc74c0692a0 0x7fc74c06b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:17:02.811 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:02.809+0000 7fc74bfff700 1 -- 172.21.15.29:0/3508698012 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fc754086bf0 con 0x7fc764101d70 2024-03-18T01:17:02.811 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:02.809+0000 7fc762ffd700 1 --2- 172.21.15.29:0/3508698012 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7fc74c0692a0 0x7fc74c06b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:17:02.811 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:02.809+0000 7fc762ffd700 1 --2- 172.21.15.29:0/3508698012 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7fc74c0692a0 0x7fc74c06b750 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fc758004890 tx=0x7fc758009e30 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:17:02.814 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:02.813+0000 7fc74bfff700 1 -- 172.21.15.29:0/3508698012 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175520 (secure 0 0 0) 0x7fc7540531e0 con 0x7fc764101d70 2024-03-18T01:17:03.005 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:03.003+0000 7fc769d9c700 1 -- 172.21.15.29:0/3508698012 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc76404bed0 con 0x7fc764101d70 2024-03-18T01:17:03.005 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:03.004+0000 7fc74bfff700 1 -- 172.21.15.29:0/3508698012 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v40) v1 ==== 74+0+11035 (secure 0 0 0) 0x7fc75400dea0 con 0x7fc764101d70 2024-03-18T01:17:03.005 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-18T01:17:03.005 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":40,"fsid":"37f51182-e4c4-11ee-95c9-87774f69a715","created":"2024-03-18T01:10:49.947064+0000","modified":"2024-03-18T01:16:42.545974+0000","last_up_change":"2024-03-18T01:16:41.540245+0000","last_in_change":"2024-03-18T01:16:23.078422+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T01:15:31.263727+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"decb820d-5c65-4317-b3f0-46522f1d1326","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":895619893},{"type":"v1","addr":"172.21.15.29:6803","nonce":895619893}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":895619893},{"type":"v1","addr":"172.21.15.29:6805","nonce":895619893}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":895619893},{"type":"v1","addr":"172.21.15.29:6809","nonce":895619893}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":895619893},{"type":"v1","addr":"172.21.15.29:6807","nonce":895619893}]},"public_addr":"172.21.15.29:6803/895619893","cluster_addr":"172.21.15.29:6805/895619893","heartbeat_back_addr":"172.21.15.29:6809/895619893","heartbeat_front_addr":"172.21.15.29:6807/895619893","state":["exists","up"]},{"osd":1,"uuid":"3b1f0295-4dd6-4bb1-9847-93bd98677db6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":2615748531},{"type":"v1","addr":"172.21.15.29:6811","nonce":2615748531}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":2615748531},{"type":"v1","addr":"172.21.15.29:6813","nonce":2615748531}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":2615748531},{"type":"v1","addr":"172.21.15.29:6817","nonce":2615748531}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":2615748531},{"type":"v1","addr":"172.21.15.29:6815","nonce":2615748531}]},"public_addr":"172.21.15.29:6811/2615748531","cluster_addr":"172.21.15.29:6813/2615748531","heartbeat_back_addr":"172.21.15.29:6817/2615748531","heartbeat_front_addr":"172.21.15.29:6815/2615748531","state":["exists","up"]},{"osd":2,"uuid":"826d30cd-7b7d-4f93-9567-c6bf6120dfe0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":576060095},{"type":"v1","addr":"172.21.15.29:6819","nonce":576060095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6820","nonce":576060095},{"type":"v1","addr":"172.21.15.29:6821","nonce":576060095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":576060095},{"type":"v1","addr":"172.21.15.29:6825","nonce":576060095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":576060095},{"type":"v1","addr":"172.21.15.29:6823","nonce":576060095}]},"public_addr":"172.21.15.29:6819/576060095","cluster_addr":"172.21.15.29:6821/576060095","heartbeat_back_addr":"172.21.15.29:6825/576060095","heartbeat_front_addr":"172.21.15.29:6823/576060095","state":["exists","up"]},{"osd":3,"uuid":"cba371da-b198-4b14-a3ac-fcf90d1a91e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":1797049062},{"type":"v1","addr":"172.21.15.90:6801","nonce":1797049062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":1797049062},{"type":"v1","addr":"172.21.15.90:6803","nonce":1797049062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":1797049062},{"type":"v1","addr":"172.21.15.90:6807","nonce":1797049062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":1797049062},{"type":"v1","addr":"172.21.15.90:6805","nonce":1797049062}]},"public_addr":"172.21.15.90:6801/1797049062","cluster_addr":"172.21.15.90:6803/1797049062","heartbeat_back_addr":"172.21.15.90:6807/1797049062","heartbeat_front_addr":"172.21.15.90:6805/1797049062","state":["exists","up"]},{"osd":4,"uuid":"c18771da-2efb-4897-a07a-ce48c94e2dee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":2675255441},{"type":"v1","addr":"172.21.15.90:6809","nonce":2675255441}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":2675255441},{"type":"v1","addr":"172.21.15.90:6811","nonce":2675255441}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":2675255441},{"type":"v1","addr":"172.21.15.90:6815","nonce":2675255441}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":2675255441},{"type":"v1","addr":"172.21.15.90:6813","nonce":2675255441}]},"public_addr":"172.21.15.90:6809/2675255441","cluster_addr":"172.21.15.90:6811/2675255441","heartbeat_back_addr":"172.21.15.90:6815/2675255441","heartbeat_front_addr":"172.21.15.90:6813/2675255441","state":["exists","up"]},{"osd":5,"uuid":"82248e69-74f9-4f46-a3fd-9224dcc8eae6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":2630668390},{"type":"v1","addr":"172.21.15.90:6817","nonce":2630668390}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6818","nonce":2630668390},{"type":"v1","addr":"172.21.15.90:6819","nonce":2630668390}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":2630668390},{"type":"v1","addr":"172.21.15.90:6823","nonce":2630668390}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6820","nonce":2630668390},{"type":"v1","addr":"172.21.15.90:6821","nonce":2630668390}]},"public_addr":"172.21.15.90:6817/2630668390","cluster_addr":"172.21.15.90:6819/2630668390","heartbeat_back_addr":"172.21.15.90:6823/2630668390","heartbeat_front_addr":"172.21.15.90:6821/2630668390","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:14:30.758603+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:14:59.452381+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:15:28.377239+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:15:51.151100+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:16:15.663658+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:16:40.030041+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.29:0/2762171838":"2024-03-19T01:13:10.926615+0000","172.21.15.29:0/3466320979":"2024-03-19T01:13:10.926615+0000","172.21.15.29:6801/2687730237":"2024-03-19T01:13:10.926615+0000","172.21.15.29:0/3723093724":"2024-03-19T01:11:59.169888+0000","172.21.15.29:6801/950856841":"2024-03-19T01:11:59.169888+0000","172.21.15.29:0/3604736621":"2024-03-19T01:11:59.169888+0000","172.21.15.29:6800/2687730237":"2024-03-19T01:13:10.926615+0000","172.21.15.29:0/3191846431":"2024-03-19T01:13:10.926615+0000","172.21.15.29:6800/1702596206":"2024-03-19T01:11:26.433683+0000","172.21.15.29:0/4093790008":"2024-03-19T01:11:26.433683+0000","172.21.15.29:6801/1702596206":"2024-03-19T01:11:26.433683+0000","172.21.15.29:0/1016951963":"2024-03-19T01:11:26.433683+0000","172.21.15.29:6800/950856841":"2024-03-19T01:11:59.169888+0000","172.21.15.29:0/2614521832":"2024-03-19T01:11:26.433683+0000","172.21.15.29:0/3826687028":"2024-03-19T01:11:59.169888+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T01:17:03.008 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:03.006+0000 7fc769d9c700 1 -- 172.21.15.29:0/3508698012 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7fc74c0692a0 msgr2=0x7fc74c06b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:17:03.008 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:03.006+0000 7fc769d9c700 1 --2- 172.21.15.29:0/3508698012 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7fc74c0692a0 0x7fc74c06b750 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fc758004890 tx=0x7fc758009e30 comp rx=0 tx=0).stop 2024-03-18T01:17:03.008 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:03.006+0000 7fc769d9c700 1 -- 172.21.15.29:0/3508698012 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc764101d70 msgr2=0x7fc76410ece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:17:03.008 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:03.006+0000 7fc769d9c700 1 --2- 172.21.15.29:0/3508698012 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc764101d70 0x7fc76410ece0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fc754002a50 tx=0x7fc7540046b0 comp rx=0 tx=0).stop 2024-03-18T01:17:03.008 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:03.006+0000 7fc769d9c700 1 -- 172.21.15.29:0/3508698012 shutdown_connections 2024-03-18T01:17:03.008 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:03.006+0000 7fc769d9c700 1 --2- 172.21.15.29:0/3508698012 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7fc74c0692a0 0x7fc74c06b750 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:17:03.008 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:03.006+0000 7fc769d9c700 1 --2- 172.21.15.29:0/3508698012 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc764101d70 0x7fc76410ece0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:17:03.008 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:03.006+0000 7fc769d9c700 1 --2- 172.21.15.29:0/3508698012 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc764102710 0x7fc76410f220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:17:03.008 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:03.006+0000 7fc769d9c700 1 -- 172.21.15.29:0/3508698012 >> 172.21.15.29:0/3508698012 conn(0x7fc764072bc0 msgr2=0x7fc7640fb080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:17:03.008 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:03.006+0000 7fc769d9c700 1 -- 172.21.15.29:0/3508698012 shutdown_connections 2024-03-18T01:17:03.008 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:03.006+0000 7fc769d9c700 1 -- 172.21.15.29:0/3508698012 wait complete. 2024-03-18T01:17:03.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:17:03 smithi090 ceph-mon[36989]: from='client.? 172.21.15.29:0/3508698012' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T01:17:03.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:17:03 smithi029 ceph-mon[29331]: from='client.? 172.21.15.29:0/3508698012' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T01:17:03.699 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-03-18T01:17:03.699 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 37f51182-e4c4-11ee-95c9-87774f69a715 -- ceph osd dump --format=json 2024-03-18T01:17:03.960 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/37f51182-e4c4-11ee-95c9-87774f69a715/mon.smithi029/config 2024-03-18T01:17:04.421 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:17:04 smithi029 ceph-mon[29331]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-18T01:17:04.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:17:04 smithi090 ceph-mon[36989]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-18T01:17:05.737 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:05.734+0000 7f4bac0a5700 1 -- 172.21.15.29:0/172355692 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4ba41037b0 msgr2=0x7f4ba40fce20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:17:05.737 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:05.734+0000 7f4bac0a5700 1 --2- 172.21.15.29:0/172355692 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4ba41037b0 0x7f4ba40fce20 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f4b94004940 tx=0x7f4b94009e30 comp rx=0 tx=0).stop 2024-03-18T01:17:05.737 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:05.734+0000 7f4bac0a5700 1 -- 172.21.15.29:0/172355692 shutdown_connections 2024-03-18T01:17:05.737 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:05.734+0000 7f4bac0a5700 1 --2- 172.21.15.29:0/172355692 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4ba41037b0 0x7f4ba40fce20 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:17:05.737 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:05.734+0000 7f4bac0a5700 1 --2- 172.21.15.29:0/172355692 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4ba4102e10 0x7f4ba41031e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:17:05.737 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:05.734+0000 7f4bac0a5700 1 -- 172.21.15.29:0/172355692 >> 172.21.15.29:0/172355692 conn(0x7f4ba40f89a0 msgr2=0x7f4ba40fadb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:17:05.737 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:05.734+0000 7f4bac0a5700 1 -- 172.21.15.29:0/172355692 shutdown_connections 2024-03-18T01:17:05.737 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:05.734+0000 7f4bac0a5700 1 -- 172.21.15.29:0/172355692 wait complete. 2024-03-18T01:17:05.738 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:05.735+0000 7f4bac0a5700 1 Processor -- start 2024-03-18T01:17:05.738 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:05.735+0000 7f4bac0a5700 1 -- start start 2024-03-18T01:17:05.738 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:05.735+0000 7f4bac0a5700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4ba4102e10 0x7f4ba4113ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:17:05.738 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:05.735+0000 7f4bac0a5700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4ba41037b0 0x7f4ba41140e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:17:05.738 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:05.735+0000 7f4bac0a5700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ba4114730 con 0x7f4ba41037b0 2024-03-18T01:17:05.738 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:05.735+0000 7f4bac0a5700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ba410db20 con 0x7f4ba4102e10 2024-03-18T01:17:05.738 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:05.735+0000 7f4ba9640700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4ba41037b0 0x7f4ba41140e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:17:05.738 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:05.735+0000 7f4ba9640700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4ba41037b0 0x7f4ba41140e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51200/0 (socket says 172.21.15.29:51200) 2024-03-18T01:17:05.738 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:05.735+0000 7f4ba9640700 1 -- 172.21.15.29:0/3023187334 learned_addr learned my addr 172.21.15.29:0/3023187334 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-18T01:17:05.738 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:05.735+0000 7f4ba9e41700 1 --2- 172.21.15.29:0/3023187334 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4ba4102e10 0x7f4ba4113ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:17:05.738 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:05.735+0000 7f4ba9640700 1 -- 172.21.15.29:0/3023187334 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4ba4102e10 msgr2=0x7f4ba4113ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:17:05.738 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:05.735+0000 7f4ba9640700 1 --2- 172.21.15.29:0/3023187334 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4ba4102e10 0x7f4ba4113ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:17:05.738 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:05.735+0000 7f4ba9640700 1 -- 172.21.15.29:0/3023187334 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b94005040 con 0x7f4ba41037b0 2024-03-18T01:17:05.738 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:05.735+0000 7f4ba9640700 1 --2- 172.21.15.29:0/3023187334 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4ba41037b0 0x7f4ba41140e0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f4b94009180 tx=0x7f4b94009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:17:05.739 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:05.736+0000 7f4b9a7fc700 1 -- 172.21.15.29:0/3023187334 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4b9400e040 con 0x7f4ba41037b0 2024-03-18T01:17:05.739 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:05.736+0000 7f4bac0a5700 1 -- 172.21.15.29:0/3023187334 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ba410dda0 con 0x7f4ba41037b0 2024-03-18T01:17:05.739 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:05.736+0000 7f4b9a7fc700 1 -- 172.21.15.29:0/3023187334 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4b94013420 con 0x7f4ba41037b0 2024-03-18T01:17:05.739 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:05.736+0000 7f4b9a7fc700 1 -- 172.21.15.29:0/3023187334 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4b9401c610 con 0x7f4ba41037b0 2024-03-18T01:17:05.739 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:05.736+0000 7f4bac0a5700 1 -- 172.21.15.29:0/3023187334 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ba410e360 con 0x7f4ba41037b0 2024-03-18T01:17:05.740 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:05.737+0000 7f4bac0a5700 1 -- 172.21.15.29:0/3023187334 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ba41052e0 con 0x7f4ba41037b0 2024-03-18T01:17:05.740 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:05.737+0000 7f4b9a7fc700 1 -- 172.21.15.29:0/3023187334 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f4b94013a00 con 0x7f4ba41037b0 2024-03-18T01:17:05.740 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:05.737+0000 7f4b9a7fc700 1 --2- 172.21.15.29:0/3023187334 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f4b900692a0 0x7f4b9006b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:17:05.740 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:05.738+0000 7f4b9a7fc700 1 -- 172.21.15.29:0/3023187334 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f4b94087f50 con 0x7f4ba41037b0 2024-03-18T01:17:05.740 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:05.738+0000 7f4ba9e41700 1 --2- 172.21.15.29:0/3023187334 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f4b900692a0 0x7f4b9006b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:17:05.740 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:05.738+0000 7f4ba9e41700 1 --2- 172.21.15.29:0/3023187334 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f4b900692a0 0x7f4b9006b750 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f4ba0007fd0 tx=0x7f4ba0009040 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:17:05.744 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:05.743+0000 7f4b9a7fc700 1 -- 172.21.15.29:0/3023187334 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175520 (secure 0 0 0) 0x7f4b940b9db0 con 0x7f4ba41037b0 2024-03-18T01:17:05.937 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:05.935+0000 7f4bac0a5700 1 -- 172.21.15.29:0/3023187334 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4ba410f1f0 con 0x7f4ba41037b0 2024-03-18T01:17:05.937 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:05.936+0000 7f4b9a7fc700 1 -- 172.21.15.29:0/3023187334 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v40) v1 ==== 74+0+11035 (secure 0 0 0) 0x7f4b94058050 con 0x7f4ba41037b0 2024-03-18T01:17:05.938 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-18T01:17:05.938 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":40,"fsid":"37f51182-e4c4-11ee-95c9-87774f69a715","created":"2024-03-18T01:10:49.947064+0000","modified":"2024-03-18T01:16:42.545974+0000","last_up_change":"2024-03-18T01:16:41.540245+0000","last_in_change":"2024-03-18T01:16:23.078422+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T01:15:31.263727+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"decb820d-5c65-4317-b3f0-46522f1d1326","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":895619893},{"type":"v1","addr":"172.21.15.29:6803","nonce":895619893}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":895619893},{"type":"v1","addr":"172.21.15.29:6805","nonce":895619893}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":895619893},{"type":"v1","addr":"172.21.15.29:6809","nonce":895619893}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":895619893},{"type":"v1","addr":"172.21.15.29:6807","nonce":895619893}]},"public_addr":"172.21.15.29:6803/895619893","cluster_addr":"172.21.15.29:6805/895619893","heartbeat_back_addr":"172.21.15.29:6809/895619893","heartbeat_front_addr":"172.21.15.29:6807/895619893","state":["exists","up"]},{"osd":1,"uuid":"3b1f0295-4dd6-4bb1-9847-93bd98677db6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":2615748531},{"type":"v1","addr":"172.21.15.29:6811","nonce":2615748531}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":2615748531},{"type":"v1","addr":"172.21.15.29:6813","nonce":2615748531}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":2615748531},{"type":"v1","addr":"172.21.15.29:6817","nonce":2615748531}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":2615748531},{"type":"v1","addr":"172.21.15.29:6815","nonce":2615748531}]},"public_addr":"172.21.15.29:6811/2615748531","cluster_addr":"172.21.15.29:6813/2615748531","heartbeat_back_addr":"172.21.15.29:6817/2615748531","heartbeat_front_addr":"172.21.15.29:6815/2615748531","state":["exists","up"]},{"osd":2,"uuid":"826d30cd-7b7d-4f93-9567-c6bf6120dfe0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":576060095},{"type":"v1","addr":"172.21.15.29:6819","nonce":576060095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6820","nonce":576060095},{"type":"v1","addr":"172.21.15.29:6821","nonce":576060095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":576060095},{"type":"v1","addr":"172.21.15.29:6825","nonce":576060095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":576060095},{"type":"v1","addr":"172.21.15.29:6823","nonce":576060095}]},"public_addr":"172.21.15.29:6819/576060095","cluster_addr":"172.21.15.29:6821/576060095","heartbeat_back_addr":"172.21.15.29:6825/576060095","heartbeat_front_addr":"172.21.15.29:6823/576060095","state":["exists","up"]},{"osd":3,"uuid":"cba371da-b198-4b14-a3ac-fcf90d1a91e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":1797049062},{"type":"v1","addr":"172.21.15.90:6801","nonce":1797049062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":1797049062},{"type":"v1","addr":"172.21.15.90:6803","nonce":1797049062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":1797049062},{"type":"v1","addr":"172.21.15.90:6807","nonce":1797049062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":1797049062},{"type":"v1","addr":"172.21.15.90:6805","nonce":1797049062}]},"public_addr":"172.21.15.90:6801/1797049062","cluster_addr":"172.21.15.90:6803/1797049062","heartbeat_back_addr":"172.21.15.90:6807/1797049062","heartbeat_front_addr":"172.21.15.90:6805/1797049062","state":["exists","up"]},{"osd":4,"uuid":"c18771da-2efb-4897-a07a-ce48c94e2dee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":2675255441},{"type":"v1","addr":"172.21.15.90:6809","nonce":2675255441}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":2675255441},{"type":"v1","addr":"172.21.15.90:6811","nonce":2675255441}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":2675255441},{"type":"v1","addr":"172.21.15.90:6815","nonce":2675255441}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":2675255441},{"type":"v1","addr":"172.21.15.90:6813","nonce":2675255441}]},"public_addr":"172.21.15.90:6809/2675255441","cluster_addr":"172.21.15.90:6811/2675255441","heartbeat_back_addr":"172.21.15.90:6815/2675255441","heartbeat_front_addr":"172.21.15.90:6813/2675255441","state":["exists","up"]},{"osd":5,"uuid":"82248e69-74f9-4f46-a3fd-9224dcc8eae6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":2630668390},{"type":"v1","addr":"172.21.15.90:6817","nonce":2630668390}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6818","nonce":2630668390},{"type":"v1","addr":"172.21.15.90:6819","nonce":2630668390}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":2630668390},{"type":"v1","addr":"172.21.15.90:6823","nonce":2630668390}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6820","nonce":2630668390},{"type":"v1","addr":"172.21.15.90:6821","nonce":2630668390}]},"public_addr":"172.21.15.90:6817/2630668390","cluster_addr":"172.21.15.90:6819/2630668390","heartbeat_back_addr":"172.21.15.90:6823/2630668390","heartbeat_front_addr":"172.21.15.90:6821/2630668390","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:14:30.758603+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:14:59.452381+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:15:28.377239+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:15:51.151100+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:16:15.663658+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:16:40.030041+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.29:0/2762171838":"2024-03-19T01:13:10.926615+0000","172.21.15.29:0/3466320979":"2024-03-19T01:13:10.926615+0000","172.21.15.29:6801/2687730237":"2024-03-19T01:13:10.926615+0000","172.21.15.29:0/3723093724":"2024-03-19T01:11:59.169888+0000","172.21.15.29:6801/950856841":"2024-03-19T01:11:59.169888+0000","172.21.15.29:0/3604736621":"2024-03-19T01:11:59.169888+0000","172.21.15.29:6800/2687730237":"2024-03-19T01:13:10.926615+0000","172.21.15.29:0/3191846431":"2024-03-19T01:13:10.926615+0000","172.21.15.29:6800/1702596206":"2024-03-19T01:11:26.433683+0000","172.21.15.29:0/4093790008":"2024-03-19T01:11:26.433683+0000","172.21.15.29:6801/1702596206":"2024-03-19T01:11:26.433683+0000","172.21.15.29:0/1016951963":"2024-03-19T01:11:26.433683+0000","172.21.15.29:6800/950856841":"2024-03-19T01:11:59.169888+0000","172.21.15.29:0/2614521832":"2024-03-19T01:11:26.433683+0000","172.21.15.29:0/3826687028":"2024-03-19T01:11:59.169888+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T01:17:05.941 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:05.938+0000 7f4bac0a5700 1 -- 172.21.15.29:0/3023187334 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f4b900692a0 msgr2=0x7f4b9006b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:17:05.941 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:05.938+0000 7f4bac0a5700 1 --2- 172.21.15.29:0/3023187334 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f4b900692a0 0x7f4b9006b750 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f4ba0007fd0 tx=0x7f4ba0009040 comp rx=0 tx=0).stop 2024-03-18T01:17:05.941 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:05.938+0000 7f4bac0a5700 1 -- 172.21.15.29:0/3023187334 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4ba41037b0 msgr2=0x7f4ba41140e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:17:05.942 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:05.938+0000 7f4bac0a5700 1 --2- 172.21.15.29:0/3023187334 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4ba41037b0 0x7f4ba41140e0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f4b94009180 tx=0x7f4b94009e30 comp rx=0 tx=0).stop 2024-03-18T01:17:05.942 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:05.938+0000 7f4bac0a5700 1 -- 172.21.15.29:0/3023187334 shutdown_connections 2024-03-18T01:17:05.942 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:05.938+0000 7f4bac0a5700 1 --2- 172.21.15.29:0/3023187334 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f4b900692a0 0x7f4b9006b750 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:17:05.942 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:05.938+0000 7f4bac0a5700 1 --2- 172.21.15.29:0/3023187334 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4ba4102e10 0x7f4ba4113ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:17:05.942 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:05.938+0000 7f4bac0a5700 1 --2- 172.21.15.29:0/3023187334 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4ba41037b0 0x7f4ba41140e0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:17:05.942 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:05.938+0000 7f4bac0a5700 1 -- 172.21.15.29:0/3023187334 >> 172.21.15.29:0/3023187334 conn(0x7f4ba40f89a0 msgr2=0x7f4ba4102220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:17:05.942 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:05.938+0000 7f4bac0a5700 1 -- 172.21.15.29:0/3023187334 shutdown_connections 2024-03-18T01:17:05.942 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:05.938+0000 7f4bac0a5700 1 -- 172.21.15.29:0/3023187334 wait complete. 2024-03-18T01:17:06.401 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:17:06 smithi029 ceph-mon[29331]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-18T01:17:06.401 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:17:06 smithi029 ceph-mon[29331]: from='client.? 172.21.15.29:0/3023187334' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T01:17:06.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:17:06 smithi090 ceph-mon[36989]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-18T01:17:06.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:17:06 smithi090 ceph-mon[36989]: from='client.? 172.21.15.29:0/3023187334' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T01:17:06.689 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 37f51182-e4c4-11ee-95c9-87774f69a715 -- ceph tell osd.0 flush_pg_stats 2024-03-18T01:17:06.689 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 37f51182-e4c4-11ee-95c9-87774f69a715 -- ceph tell osd.1 flush_pg_stats 2024-03-18T01:17:06.689 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 37f51182-e4c4-11ee-95c9-87774f69a715 -- ceph tell osd.2 flush_pg_stats 2024-03-18T01:17:06.690 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 37f51182-e4c4-11ee-95c9-87774f69a715 -- ceph tell osd.3 flush_pg_stats 2024-03-18T01:17:06.690 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 37f51182-e4c4-11ee-95c9-87774f69a715 -- ceph tell osd.4 flush_pg_stats 2024-03-18T01:17:06.690 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 37f51182-e4c4-11ee-95c9-87774f69a715 -- ceph tell osd.5 flush_pg_stats 2024-03-18T01:17:06.990 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/37f51182-e4c4-11ee-95c9-87774f69a715/mon.smithi029/config 2024-03-18T01:17:06.994 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/37f51182-e4c4-11ee-95c9-87774f69a715/mon.smithi029/config 2024-03-18T01:17:08.427 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:17:08 smithi090 ceph-mon[36989]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-18T01:17:08.432 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:17:08 smithi029 ceph-mon[29331]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-18T01:17:09.458 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:09.456+0000 7f919fad1700 1 -- 172.21.15.29:0/3049749173 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f91980fded0 msgr2=0x7f91980fe2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:17:09.458 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:09.456+0000 7f919fad1700 1 --2- 172.21.15.29:0/3049749173 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f91980fded0 0x7f91980fe2a0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f9190009ce0 tx=0x7f9190004d10 comp rx=0 tx=0).stop 2024-03-18T01:17:09.458 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:09.457+0000 7f919fad1700 1 -- 172.21.15.29:0/3049749173 shutdown_connections 2024-03-18T01:17:09.458 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:09.457+0000 7f919fad1700 1 --2- 172.21.15.29:0/3049749173 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f91980fe870 0x7f9198106f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:17:09.458 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:09.457+0000 7f919fad1700 1 --2- 172.21.15.29:0/3049749173 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f91980fded0 0x7f91980fe2a0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:17:09.458 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:09.457+0000 7f919fad1700 1 -- 172.21.15.29:0/3049749173 >> 172.21.15.29:0/3049749173 conn(0x7f9198072bc0 msgr2=0x7f9198072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:17:09.458 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:09.457+0000 7f919fad1700 1 -- 172.21.15.29:0/3049749173 shutdown_connections 2024-03-18T01:17:09.458 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:09.457+0000 7f919fad1700 1 -- 172.21.15.29:0/3049749173 wait complete. 2024-03-18T01:17:09.458 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:09.457+0000 7f919fad1700 1 Processor -- start 2024-03-18T01:17:09.458 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:09.458+0000 7f919fad1700 1 -- start start 2024-03-18T01:17:09.460 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:09.458+0000 7f919fad1700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f91980fe870 0x7f9198113cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:17:09.460 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:09.458+0000 7f919fad1700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9198114210 0x7f919810dc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:17:09.460 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:09.458+0000 7f919fad1700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9198114790 con 0x7f9198114210 2024-03-18T01:17:09.460 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:09.458+0000 7f919fad1700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9198114900 con 0x7f91980fe870 2024-03-18T01:17:09.460 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:09.458+0000 7f919d06c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9198114210 0x7f919810dc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:17:09.460 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:09.458+0000 7f919d06c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9198114210 0x7f919810dc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51212/0 (socket says 172.21.15.29:51212) 2024-03-18T01:17:09.461 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:09.458+0000 7f919d86d700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f91980fe870 0x7f9198113cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:17:09.461 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:09.458+0000 7f919d06c700 1 -- 172.21.15.29:0/3079187796 learned_addr learned my addr 172.21.15.29:0/3079187796 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-18T01:17:09.461 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:09.458+0000 7f919d86d700 1 -- 172.21.15.29:0/3079187796 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9198114210 msgr2=0x7f919810dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:17:09.461 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:09.458+0000 7f919d86d700 1 --2- 172.21.15.29:0/3079187796 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9198114210 0x7f919810dc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:17:09.461 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:09.458+0000 7f919d86d700 1 -- 172.21.15.29:0/3079187796 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9190009940 con 0x7f91980fe870 2024-03-18T01:17:09.461 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:09.458+0000 7f919d06c700 1 --2- 172.21.15.29:0/3079187796 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9198114210 0x7f919810dc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T01:17:09.461 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:09.458+0000 7f919d86d700 1 --2- 172.21.15.29:0/3079187796 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f91980fe870 0x7f9198113cd0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f9190008000 tx=0x7f9190004d10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:17:09.461 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:09.459+0000 7f91967fc700 1 -- 172.21.15.29:0/3079187796 <== mon.1 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f919000e040 con 0x7f91980fe870 2024-03-18T01:17:09.461 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:09.459+0000 7f919fad1700 1 -- 172.21.15.29:0/3079187796 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f919810e230 con 0x7f91980fe870 2024-03-18T01:17:09.461 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:09.459+0000 7f91967fc700 1 -- 172.21.15.29:0/3079187796 <== mon.1 v2:172.21.15.90:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9190007e80 con 0x7f91980fe870 2024-03-18T01:17:09.461 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:09.459+0000 7f91967fc700 1 -- 172.21.15.29:0/3079187796 <== mon.1 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f919001c660 con 0x7f91980fe870 2024-03-18T01:17:09.461 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:09.459+0000 7f919fad1700 1 -- 172.21.15.29:0/3079187796 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f919810e740 con 0x7f91980fe870 2024-03-18T01:17:09.461 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:09.460+0000 7f919fad1700 1 -- 172.21.15.29:0/3079187796 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f919805e6c0 con 0x7f91980fe870 2024-03-18T01:17:09.461 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:09.460+0000 7f91967fc700 1 -- 172.21.15.29:0/3079187796 <== mon.1 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f9190012620 con 0x7f91980fe870 2024-03-18T01:17:09.463 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:09.461+0000 7f91967fc700 1 --2- 172.21.15.29:0/3079187796 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f91800692b0 0x7f918006b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:17:09.463 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:09.461+0000 7f91967fc700 1 -- 172.21.15.29:0/3079187796 <== mon.1 v2:172.21.15.90:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f9190088c60 con 0x7f91980fe870 2024-03-18T01:17:09.463 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:09.461+0000 7f919d06c700 1 --2- 172.21.15.29:0/3079187796 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f91800692b0 0x7f918006b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:17:09.463 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:09.461+0000 7f91967fc700 1 --2- 172.21.15.29:0/3079187796 >> [v2:172.21.15.29:6818/576060095,v1:172.21.15.29:6819/576060095] conn(0x7f918006ef80 0x7f9180071390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:17:09.463 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:09.461+0000 7f91967fc700 1 -- 172.21.15.29:0/3079187796 --> [v2:172.21.15.29:6818/576060095,v1:172.21.15.29:6819/576060095] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9180071a40 con 0x7f918006ef80 2024-03-18T01:17:09.463 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:09.461+0000 7f91967fc700 1 -- 172.21.15.29:0/3079187796 <== mon.1 v2:172.21.15.90:3300/0 6 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7f919008d220 con 0x7f91980fe870 2024-03-18T01:17:09.463 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:09.461+0000 7f919e06e700 1 --2- 172.21.15.29:0/3079187796 >> [v2:172.21.15.29:6818/576060095,v1:172.21.15.29:6819/576060095] conn(0x7f918006ef80 0x7f9180071390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:17:09.463 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:09.461+0000 7f919d06c700 1 --2- 172.21.15.29:0/3079187796 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f91800692b0 0x7f918006b760 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f918c0041d0 tx=0x7f918c00b040 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:17:09.463 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:09.461+0000 7f919e06e700 1 --2- 172.21.15.29:0/3079187796 >> [v2:172.21.15.29:6818/576060095,v1:172.21.15.29:6819/576060095] conn(0x7f918006ef80 0x7f9180071390 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:17:09.463 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:09.463+0000 7f91967fc700 1 -- 172.21.15.29:0/3079187796 <== osd.2 v2:172.21.15.29:6818/576060095 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7f9180071a40 con 0x7f918006ef80 2024-03-18T01:17:09.473 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/37f51182-e4c4-11ee-95c9-87774f69a715/mon.smithi029/config 2024-03-18T01:17:09.483 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:09.483+0000 7f919fad1700 1 -- 172.21.15.29:0/3079187796 --> [v2:172.21.15.29:6818/576060095,v1:172.21.15.29:6819/576060095] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f9198065df0 con 0x7f918006ef80 2024-03-18T01:17:09.484 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:09.483+0000 7f91967fc700 1 -- 172.21.15.29:0/3079187796 <== osd.2 v2:172.21.15.29:6818/576060095 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f9198065df0 con 0x7f918006ef80 2024-03-18T01:17:09.484 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:09.483+0000 7f919fad1700 1 -- 172.21.15.29:0/3079187796 >> [v2:172.21.15.29:6818/576060095,v1:172.21.15.29:6819/576060095] conn(0x7f918006ef80 msgr2=0x7f9180071390 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:17:09.485 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:09.483+0000 7f919fad1700 1 --2- 172.21.15.29:0/3079187796 >> [v2:172.21.15.29:6818/576060095,v1:172.21.15.29:6819/576060095] conn(0x7f918006ef80 0x7f9180071390 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:17:09.485 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:09.484+0000 7f919fad1700 1 -- 172.21.15.29:0/3079187796 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f91800692b0 msgr2=0x7f918006b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:17:09.485 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:09.484+0000 7f919fad1700 1 --2- 172.21.15.29:0/3079187796 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f91800692b0 0x7f918006b760 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f918c0041d0 tx=0x7f918c00b040 comp rx=0 tx=0).stop 2024-03-18T01:17:09.485 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:09.484+0000 7f919fad1700 1 -- 172.21.15.29:0/3079187796 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f91980fe870 msgr2=0x7f9198113cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:17:09.485 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:09.484+0000 7f919fad1700 1 --2- 172.21.15.29:0/3079187796 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f91980fe870 0x7f9198113cd0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f9190008000 tx=0x7f9190004d10 comp rx=0 tx=0).stop 2024-03-18T01:17:09.485 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:09.484+0000 7f919fad1700 1 -- 172.21.15.29:0/3079187796 shutdown_connections 2024-03-18T01:17:09.485 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:09.484+0000 7f919fad1700 1 --2- 172.21.15.29:0/3079187796 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f91800692b0 0x7f918006b760 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:17:09.485 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:09.484+0000 7f919fad1700 1 --2- 172.21.15.29:0/3079187796 >> [v2:172.21.15.29:6818/576060095,v1:172.21.15.29:6819/576060095] conn(0x7f918006ef80 0x7f9180071390 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:17:09.485 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:09.484+0000 7f919fad1700 1 --2- 172.21.15.29:0/3079187796 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f91980fe870 0x7f9198113cd0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:17:09.485 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:09.484+0000 7f919fad1700 1 --2- 172.21.15.29:0/3079187796 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9198114210 0x7f919810dc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:17:09.486 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:09.484+0000 7f919fad1700 1 -- 172.21.15.29:0/3079187796 >> 172.21.15.29:0/3079187796 conn(0x7f9198072bc0 msgr2=0x7f9198106b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:17:09.486 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:09.484+0000 7f919fad1700 1 -- 172.21.15.29:0/3079187796 shutdown_connections 2024-03-18T01:17:09.486 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:09.484+0000 7f919fad1700 1 -- 172.21.15.29:0/3079187796 wait complete. 2024-03-18T01:17:10.231 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/37f51182-e4c4-11ee-95c9-87774f69a715/mon.smithi029/config 2024-03-18T01:17:10.319 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:17:10 smithi029 ceph-mon[29331]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-18T01:17:10.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:17:10 smithi090 ceph-mon[36989]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-18T01:17:10.694 INFO:teuthology.orchestra.run.smithi029.stdout:81604378647 2024-03-18T01:17:10.695 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 37f51182-e4c4-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.2 2024-03-18T01:17:10.922 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/37f51182-e4c4-11ee-95c9-87774f69a715/mon.smithi029/config 2024-03-18T01:17:10.923 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/37f51182-e4c4-11ee-95c9-87774f69a715/mon.smithi029/config 2024-03-18T01:17:11.191 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:17:11 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-18T01:17:11.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:17:11 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-18T01:17:12.385 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:17:12 smithi029 ceph-mon[29331]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-18T01:17:12.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:17:12 smithi090 ceph-mon[36989]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-18T01:17:13.834 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:11.420+0000 7fb2c924b700 1 -- 172.21.15.29:0/1600476209 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb2c41026c0 msgr2=0x7fb2c40fbd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:17:13.834 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:11.420+0000 7fb2c924b700 1 --2- 172.21.15.29:0/1600476209 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb2c41026c0 0x7fb2c40fbd30 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fb2b4004940 tx=0x7fb2b4009e30 comp rx=0 tx=0).stop 2024-03-18T01:17:13.834 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:11.420+0000 7fb2c924b700 1 -- 172.21.15.29:0/1600476209 shutdown_connections 2024-03-18T01:17:13.834 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:11.420+0000 7fb2c924b700 1 --2- 172.21.15.29:0/1600476209 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb2c41026c0 0x7fb2c40fbd30 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:17:13.834 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:11.420+0000 7fb2c924b700 1 --2- 172.21.15.29:0/1600476209 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb2c4101d20 0x7fb2c41020f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:17:13.834 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:11.420+0000 7fb2c924b700 1 -- 172.21.15.29:0/1600476209 >> 172.21.15.29:0/1600476209 conn(0x7fb2c40f78d0 msgr2=0x7fb2c40f9ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:17:13.834 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:11.420+0000 7fb2c924b700 1 -- 172.21.15.29:0/1600476209 shutdown_connections 2024-03-18T01:17:13.835 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:11.420+0000 7fb2c924b700 1 -- 172.21.15.29:0/1600476209 wait complete. 2024-03-18T01:17:13.835 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:11.421+0000 7fb2c924b700 1 Processor -- start 2024-03-18T01:17:13.835 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:11.421+0000 7fb2c924b700 1 -- start start 2024-03-18T01:17:13.835 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:11.421+0000 7fb2c924b700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb2c4101d20 0x7fb2c410ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:17:13.835 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:11.421+0000 7fb2c924b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb2c41026c0 0x7fb2c410f210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:17:13.835 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:11.421+0000 7fb2c924b700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2c410f7e0 con 0x7fb2c41026c0 2024-03-18T01:17:13.835 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:11.421+0000 7fb2c924b700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2c410f920 con 0x7fb2c4101d20 2024-03-18T01:17:13.835 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:11.421+0000 7fb2c259c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb2c41026c0 0x7fb2c410f210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:17:13.835 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:11.421+0000 7fb2c259c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb2c41026c0 0x7fb2c410f210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51234/0 (socket says 172.21.15.29:51234) 2024-03-18T01:17:13.835 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:11.421+0000 7fb2c259c700 1 -- 172.21.15.29:0/4270176387 learned_addr learned my addr 172.21.15.29:0/4270176387 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-18T01:17:13.835 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:11.421+0000 7fb2c2d9d700 1 --2- 172.21.15.29:0/4270176387 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb2c4101d20 0x7fb2c410ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:17:13.835 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:11.421+0000 7fb2c259c700 1 -- 172.21.15.29:0/4270176387 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb2c4101d20 msgr2=0x7fb2c410ecd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:17:13.835 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:11.421+0000 7fb2c259c700 1 --2- 172.21.15.29:0/4270176387 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb2c4101d20 0x7fb2c410ecd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:17:13.836 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:11.421+0000 7fb2c259c700 1 -- 172.21.15.29:0/4270176387 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2b4005040 con 0x7fb2c41026c0 2024-03-18T01:17:13.836 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:11.422+0000 7fb2c259c700 1 --2- 172.21.15.29:0/4270176387 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb2c41026c0 0x7fb2c410f210 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fb2b4009100 tx=0x7fb2b4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:17:13.836 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:11.422+0000 7fb2ab7fe700 1 -- 172.21.15.29:0/4270176387 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb2b400e040 con 0x7fb2c41026c0 2024-03-18T01:17:13.836 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:11.422+0000 7fb2ab7fe700 1 -- 172.21.15.29:0/4270176387 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb2b4009af0 con 0x7fb2c41026c0 2024-03-18T01:17:13.836 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:11.422+0000 7fb2c924b700 1 -- 172.21.15.29:0/4270176387 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2c4117be0 con 0x7fb2c41026c0 2024-03-18T01:17:13.836 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:11.422+0000 7fb2ab7fe700 1 -- 172.21.15.29:0/4270176387 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb2b401c3f0 con 0x7fb2c41026c0 2024-03-18T01:17:13.836 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:11.422+0000 7fb2c924b700 1 -- 172.21.15.29:0/4270176387 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2c4118170 con 0x7fb2c41026c0 2024-03-18T01:17:13.837 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:11.423+0000 7fb2c924b700 1 -- 172.21.15.29:0/4270176387 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fb2c40f9240 con 0x7fb2c41026c0 2024-03-18T01:17:13.837 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:11.423+0000 7fb2ab7fe700 1 -- 172.21.15.29:0/4270176387 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7fb2b401c6b0 con 0x7fb2c41026c0 2024-03-18T01:17:13.837 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:11.424+0000 7fb2ab7fe700 1 --2- 172.21.15.29:0/4270176387 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7fb2ac0692a0 0x7fb2ac06b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:17:13.837 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:11.424+0000 7fb2ab7fe700 1 -- 172.21.15.29:0/4270176387 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fb2b4088070 con 0x7fb2c41026c0 2024-03-18T01:17:13.837 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:11.424+0000 7fb2c2d9d700 1 --2- 172.21.15.29:0/4270176387 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7fb2ac0692a0 0x7fb2ac06b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:17:13.837 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:11.424+0000 7fb2ab7fe700 1 --2- 172.21.15.29:0/4270176387 >> [v2:172.21.15.90:6800/1797049062,v1:172.21.15.90:6801/1797049062] conn(0x7fb2ac06ef70 0x7fb2ac071380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:17:13.837 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:11.424+0000 7fb2ab7fe700 1 -- 172.21.15.29:0/4270176387 --> [v2:172.21.15.90:6800/1797049062,v1:172.21.15.90:6801/1797049062] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb2ac071a30 con 0x7fb2ac06ef70 2024-03-18T01:17:13.837 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:11.424+0000 7fb2ab7fe700 1 -- 172.21.15.29:0/4270176387 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb2b408a6c0 con 0x7fb2c41026c0 2024-03-18T01:17:13.837 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:11.424+0000 7fb2c2d9d700 1 --2- 172.21.15.29:0/4270176387 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7fb2ac0692a0 0x7fb2ac06b750 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fb2b8009b60 tx=0x7fb2b80091a0 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:17:13.837 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:11.424+0000 7fb2c359e700 1 --2- 172.21.15.29:0/4270176387 >> [v2:172.21.15.90:6800/1797049062,v1:172.21.15.90:6801/1797049062] conn(0x7fb2ac06ef70 0x7fb2ac071380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:17:13.837 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:11.425+0000 7fb2c359e700 1 --2- 172.21.15.29:0/4270176387 >> [v2:172.21.15.90:6800/1797049062,v1:172.21.15.90:6801/1797049062] conn(0x7fb2ac06ef70 0x7fb2ac071380 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:17:13.837 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:11.426+0000 7fb2ab7fe700 1 -- 172.21.15.29:0/4270176387 <== osd.3 v2:172.21.15.90:6800/1797049062 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7fb2ac071a30 con 0x7fb2ac06ef70 2024-03-18T01:17:13.837 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:11.441+0000 7fb2c924b700 1 -- 172.21.15.29:0/4270176387 --> [v2:172.21.15.90:6800/1797049062,v1:172.21.15.90:6801/1797049062] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fb2c40fd470 con 0x7fb2ac06ef70 2024-03-18T01:17:13.837 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:11.441+0000 7fb2ab7fe700 1 -- 172.21.15.29:0/4270176387 <== osd.3 v2:172.21.15.90:6800/1797049062 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7fb2c40fd470 con 0x7fb2ac06ef70 2024-03-18T01:17:13.838 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:11.441+0000 7fb2c924b700 1 -- 172.21.15.29:0/4270176387 >> [v2:172.21.15.90:6800/1797049062,v1:172.21.15.90:6801/1797049062] conn(0x7fb2ac06ef70 msgr2=0x7fb2ac071380 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:17:13.838 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:11.441+0000 7fb2c924b700 1 --2- 172.21.15.29:0/4270176387 >> [v2:172.21.15.90:6800/1797049062,v1:172.21.15.90:6801/1797049062] conn(0x7fb2ac06ef70 0x7fb2ac071380 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:17:13.838 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:11.442+0000 7fb2c924b700 1 -- 172.21.15.29:0/4270176387 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7fb2ac0692a0 msgr2=0x7fb2ac06b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:17:13.838 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:11.442+0000 7fb2c924b700 1 --2- 172.21.15.29:0/4270176387 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7fb2ac0692a0 0x7fb2ac06b750 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fb2b8009b60 tx=0x7fb2b80091a0 comp rx=0 tx=0).stop 2024-03-18T01:17:13.838 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:11.442+0000 7fb2c924b700 1 -- 172.21.15.29:0/4270176387 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb2c41026c0 msgr2=0x7fb2c410f210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:17:13.838 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:11.442+0000 7fb2c924b700 1 --2- 172.21.15.29:0/4270176387 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb2c41026c0 0x7fb2c410f210 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fb2b4009100 tx=0x7fb2b4009e30 comp rx=0 tx=0).stop 2024-03-18T01:17:13.838 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:11.442+0000 7fb2c924b700 1 -- 172.21.15.29:0/4270176387 shutdown_connections 2024-03-18T01:17:13.838 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:11.442+0000 7fb2c924b700 1 --2- 172.21.15.29:0/4270176387 >> [v2:172.21.15.90:6800/1797049062,v1:172.21.15.90:6801/1797049062] conn(0x7fb2ac06ef70 0x7fb2ac071380 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:17:13.838 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:11.442+0000 7fb2c924b700 1 --2- 172.21.15.29:0/4270176387 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7fb2ac0692a0 0x7fb2ac06b750 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:17:13.838 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:11.442+0000 7fb2c924b700 1 --2- 172.21.15.29:0/4270176387 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb2c4101d20 0x7fb2c410ecd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:17:13.838 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:11.442+0000 7fb2c924b700 1 --2- 172.21.15.29:0/4270176387 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb2c41026c0 0x7fb2c410f210 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:17:13.838 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:11.442+0000 7fb2c924b700 1 -- 172.21.15.29:0/4270176387 >> 172.21.15.29:0/4270176387 conn(0x7fb2c40f78d0 msgr2=0x7fb2c40f8870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:17:13.838 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:11.442+0000 7fb2c924b700 1 -- 172.21.15.29:0/4270176387 shutdown_connections 2024-03-18T01:17:13.838 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:11.442+0000 7fb2c924b700 1 -- 172.21.15.29:0/4270176387 wait complete. 2024-03-18T01:17:14.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:17:14 smithi090 ceph-mon[36989]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-18T01:17:14.543 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:17:14 smithi029 ceph-mon[29331]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-18T01:17:14.575 INFO:teuthology.orchestra.run.smithi029.stdout:111669149714 2024-03-18T01:17:14.576 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 37f51182-e4c4-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.3 2024-03-18T01:17:16.319 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:17:16 smithi029 ceph-mon[29331]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-18T01:17:16.384 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/37f51182-e4c4-11ee-95c9-87774f69a715/mon.smithi029/config 2024-03-18T01:17:16.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:17:16 smithi090 ceph-mon[36989]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-18T01:17:17.189 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:17.085+0000 7f98c1ec4700 1 -- 172.21.15.29:0/3336153795 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f98bc102de0 msgr2=0x7f98bc1031b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:17:17.189 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:17.085+0000 7f98c1ec4700 1 --2- 172.21.15.29:0/3336153795 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f98bc102de0 0x7f98bc1031b0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f98ac009ce0 tx=0x7f98ac004d10 comp rx=0 tx=0).stop 2024-03-18T01:17:17.190 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:17.085+0000 7f98c1ec4700 1 -- 172.21.15.29:0/3336153795 shutdown_connections 2024-03-18T01:17:17.190 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:17.085+0000 7f98c1ec4700 1 --2- 172.21.15.29:0/3336153795 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f98bc103780 0x7f98bc0fcdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:17:17.190 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:17.085+0000 7f98c1ec4700 1 --2- 172.21.15.29:0/3336153795 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f98bc102de0 0x7f98bc1031b0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:17:17.190 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:17.085+0000 7f98c1ec4700 1 -- 172.21.15.29:0/3336153795 >> 172.21.15.29:0/3336153795 conn(0x7f98bc0f8950 msgr2=0x7f98bc0fad60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:17:17.190 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:17.085+0000 7f98c1ec4700 1 -- 172.21.15.29:0/3336153795 shutdown_connections 2024-03-18T01:17:17.190 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:17.085+0000 7f98c1ec4700 1 -- 172.21.15.29:0/3336153795 wait complete. 2024-03-18T01:17:17.190 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:17.086+0000 7f98c1ec4700 1 Processor -- start 2024-03-18T01:17:17.190 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:17.086+0000 7f98c1ec4700 1 -- start start 2024-03-18T01:17:17.190 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:17.086+0000 7f98c1ec4700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f98bc102de0 0x7f98bc0702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:17:17.190 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:17.086+0000 7f98c1ec4700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f98bc103780 0x7f98bc070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:17:17.190 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:17.086+0000 7f98c1ec4700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98bc0722a0 con 0x7f98bc103780 2024-03-18T01:17:17.190 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:17.086+0000 7f98c1ec4700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98bc070d40 con 0x7f98bc102de0 2024-03-18T01:17:17.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:17.086+0000 7f98baffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f98bc103780 0x7f98bc070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:17:17.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:17.086+0000 7f98baffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f98bc103780 0x7f98bc070800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60348/0 (socket says 172.21.15.29:60348) 2024-03-18T01:17:17.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:17.086+0000 7f98baffd700 1 -- 172.21.15.29:0/1828580800 learned_addr learned my addr 172.21.15.29:0/1828580800 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-18T01:17:17.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:17.087+0000 7f98bb7fe700 1 --2- 172.21.15.29:0/1828580800 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f98bc102de0 0x7f98bc0702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:17:17.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:17.087+0000 7f98baffd700 1 -- 172.21.15.29:0/1828580800 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f98bc102de0 msgr2=0x7f98bc0702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:17:17.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:17.087+0000 7f98baffd700 1 --2- 172.21.15.29:0/1828580800 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f98bc102de0 0x7f98bc0702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:17:17.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:17.087+0000 7f98baffd700 1 -- 172.21.15.29:0/1828580800 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98ac009940 con 0x7f98bc103780 2024-03-18T01:17:17.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:17.087+0000 7f98baffd700 1 --2- 172.21.15.29:0/1828580800 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f98bc103780 0x7f98bc070800 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f98b000b790 tx=0x7f98b0009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:17:17.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:17.087+0000 7f98a3fff700 1 -- 172.21.15.29:0/1828580800 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f98b000f6f0 con 0x7f98bc103780 2024-03-18T01:17:17.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:17.087+0000 7f98c1ec4700 1 -- 172.21.15.29:0/1828580800 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98bc071020 con 0x7f98bc103780 2024-03-18T01:17:17.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:17.087+0000 7f98c1ec4700 1 -- 172.21.15.29:0/1828580800 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98bc11a350 con 0x7f98bc103780 2024-03-18T01:17:17.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:17.088+0000 7f98a3fff700 1 -- 172.21.15.29:0/1828580800 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f98b000fd20 con 0x7f98bc103780 2024-03-18T01:17:17.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:17.088+0000 7f98a3fff700 1 -- 172.21.15.29:0/1828580800 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f98b00103a0 con 0x7f98bc103780 2024-03-18T01:17:17.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:17.089+0000 7f98c1ec4700 1 -- 172.21.15.29:0/1828580800 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f98a8001000 con 0x7f98bc103780 2024-03-18T01:17:17.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:17.089+0000 7f98a3fff700 1 -- 172.21.15.29:0/1828580800 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f98b0010500 con 0x7f98bc103780 2024-03-18T01:17:17.192 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:17.089+0000 7f98a3fff700 1 --2- 172.21.15.29:0/1828580800 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f98a40692a0 0x7f98a406b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:17:17.192 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:17.089+0000 7f98a3fff700 1 -- 172.21.15.29:0/1828580800 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f98b0086aa0 con 0x7f98bc103780 2024-03-18T01:17:17.192 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:17.090+0000 7f98bb7fe700 1 --2- 172.21.15.29:0/1828580800 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f98a40692a0 0x7f98a406b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:17:17.192 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:17.090+0000 7f98a3fff700 1 --2- 172.21.15.29:0/1828580800 >> [v2:172.21.15.29:6810/2615748531,v1:172.21.15.29:6811/2615748531] conn(0x7f98a406ef70 0x7f98a4071380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:17:17.192 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:17.090+0000 7f98a3fff700 1 -- 172.21.15.29:0/1828580800 --> [v2:172.21.15.29:6810/2615748531,v1:172.21.15.29:6811/2615748531] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f98a4071a30 con 0x7f98a406ef70 2024-03-18T01:17:17.192 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:17.090+0000 7f98a3fff700 1 -- 172.21.15.29:0/1828580800 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7f98b008b220 con 0x7f98bc103780 2024-03-18T01:17:17.192 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:17.090+0000 7f98bbfff700 1 --2- 172.21.15.29:0/1828580800 >> [v2:172.21.15.29:6810/2615748531,v1:172.21.15.29:6811/2615748531] conn(0x7f98a406ef70 0x7f98a4071380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:17:17.192 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:17.090+0000 7f98bb7fe700 1 --2- 172.21.15.29:0/1828580800 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f98a40692a0 0x7f98a406b750 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f98ac00b740 tx=0x7f98ac004d10 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:17:17.192 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:17.090+0000 7f98bbfff700 1 --2- 172.21.15.29:0/1828580800 >> [v2:172.21.15.29:6810/2615748531,v1:172.21.15.29:6811/2615748531] conn(0x7f98a406ef70 0x7f98a4071380 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:17:17.192 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:17.091+0000 7f98a3fff700 1 -- 172.21.15.29:0/1828580800 <== osd.1 v2:172.21.15.29:6810/2615748531 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7f98a4071a30 con 0x7f98a406ef70 2024-03-18T01:17:17.192 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:17.106+0000 7f98c1ec4700 1 -- 172.21.15.29:0/1828580800 --> [v2:172.21.15.29:6810/2615748531,v1:172.21.15.29:6811/2615748531] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f98a8002d60 con 0x7f98a406ef70 2024-03-18T01:17:17.192 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:17.107+0000 7f98a3fff700 1 -- 172.21.15.29:0/1828580800 <== osd.1 v2:172.21.15.29:6810/2615748531 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f98a8002d60 con 0x7f98a406ef70 2024-03-18T01:17:17.192 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:17.107+0000 7f98c1ec4700 1 -- 172.21.15.29:0/1828580800 >> [v2:172.21.15.29:6810/2615748531,v1:172.21.15.29:6811/2615748531] conn(0x7f98a406ef70 msgr2=0x7f98a4071380 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:17:17.192 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:17.107+0000 7f98c1ec4700 1 --2- 172.21.15.29:0/1828580800 >> [v2:172.21.15.29:6810/2615748531,v1:172.21.15.29:6811/2615748531] conn(0x7f98a406ef70 0x7f98a4071380 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:17:17.192 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:17.107+0000 7f98c1ec4700 1 -- 172.21.15.29:0/1828580800 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f98a40692a0 msgr2=0x7f98a406b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:17:17.193 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:17.107+0000 7f98c1ec4700 1 --2- 172.21.15.29:0/1828580800 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f98a40692a0 0x7f98a406b750 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f98ac00b740 tx=0x7f98ac004d10 comp rx=0 tx=0).stop 2024-03-18T01:17:17.193 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:17.107+0000 7f98c1ec4700 1 -- 172.21.15.29:0/1828580800 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f98bc103780 msgr2=0x7f98bc070800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:17:17.193 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:17.107+0000 7f98c1ec4700 1 --2- 172.21.15.29:0/1828580800 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f98bc103780 0x7f98bc070800 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f98b000b790 tx=0x7f98b0009220 comp rx=0 tx=0).stop 2024-03-18T01:17:17.193 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:17.107+0000 7f98c1ec4700 1 -- 172.21.15.29:0/1828580800 shutdown_connections 2024-03-18T01:17:17.193 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:17.107+0000 7f98c1ec4700 1 --2- 172.21.15.29:0/1828580800 >> [v2:172.21.15.29:6810/2615748531,v1:172.21.15.29:6811/2615748531] conn(0x7f98a406ef70 0x7f98a4071380 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:17:17.193 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:17.107+0000 7f98c1ec4700 1 --2- 172.21.15.29:0/1828580800 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f98a40692a0 0x7f98a406b750 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:17:17.193 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:17.107+0000 7f98c1ec4700 1 --2- 172.21.15.29:0/1828580800 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f98bc102de0 0x7f98bc0702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:17:17.193 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:17.107+0000 7f98c1ec4700 1 --2- 172.21.15.29:0/1828580800 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f98bc103780 0x7f98bc070800 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:17:17.193 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:17.107+0000 7f98c1ec4700 1 -- 172.21.15.29:0/1828580800 >> 172.21.15.29:0/1828580800 conn(0x7f98bc0f8950 msgr2=0x7f98bc107380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:17:17.193 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:17.107+0000 7f98c1ec4700 1 -- 172.21.15.29:0/1828580800 shutdown_connections 2024-03-18T01:17:17.193 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:17.107+0000 7f98c1ec4700 1 -- 172.21.15.29:0/1828580800 wait complete. 2024-03-18T01:17:17.431 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/37f51182-e4c4-11ee-95c9-87774f69a715/mon.smithi029/config 2024-03-18T01:17:18.440 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:17:18 smithi029 ceph-mon[29331]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-18T01:17:18.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:17:18 smithi090 ceph-mon[36989]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-18T01:17:19.023 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:17.194+0000 7fdedfb14700 1 -- 172.21.15.29:0/2998150917 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fded81037b0 msgr2=0x7fded80fce20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:17:19.023 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:17.194+0000 7fdedfb14700 1 --2- 172.21.15.29:0/2998150917 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fded81037b0 0x7fded80fce20 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fdec8004940 tx=0x7fdec8009e30 comp rx=0 tx=0).stop 2024-03-18T01:17:19.024 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:17.194+0000 7fdedfb14700 1 -- 172.21.15.29:0/2998150917 shutdown_connections 2024-03-18T01:17:19.024 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:17.194+0000 7fdedfb14700 1 --2- 172.21.15.29:0/2998150917 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fded81037b0 0x7fded80fce20 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:17:19.024 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:17.194+0000 7fdedfb14700 1 --2- 172.21.15.29:0/2998150917 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fded8102e10 0x7fded81031e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:17:19.024 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:17.194+0000 7fdedfb14700 1 -- 172.21.15.29:0/2998150917 >> 172.21.15.29:0/2998150917 conn(0x7fded80f89a0 msgr2=0x7fded80fadb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:17:19.024 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:17.194+0000 7fdedfb14700 1 -- 172.21.15.29:0/2998150917 shutdown_connections 2024-03-18T01:17:19.024 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:17.194+0000 7fdedfb14700 1 -- 172.21.15.29:0/2998150917 wait complete. 2024-03-18T01:17:19.024 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:17.195+0000 7fdedfb14700 1 Processor -- start 2024-03-18T01:17:19.024 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:17.195+0000 7fdedfb14700 1 -- start start 2024-03-18T01:17:19.024 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:17.195+0000 7fdedfb14700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fded8102e10 0x7fded8113b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:17:19.024 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:17.195+0000 7fdedfb14700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fded81037b0 0x7fded81140c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:17:19.024 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:17.195+0000 7fdedfb14700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fded8114710 con 0x7fded81037b0 2024-03-18T01:17:19.024 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:17.195+0000 7fdedfb14700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fded810db20 con 0x7fded8102e10 2024-03-18T01:17:19.024 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:17.195+0000 7fdedd0af700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fded81037b0 0x7fded81140c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:17:19.025 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:17.195+0000 7fdedd0af700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fded81037b0 0x7fded81140c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60370/0 (socket says 172.21.15.29:60370) 2024-03-18T01:17:19.025 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:17.195+0000 7fdedd0af700 1 -- 172.21.15.29:0/2877374839 learned_addr learned my addr 172.21.15.29:0/2877374839 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-18T01:17:19.025 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:17.195+0000 7fdedd8b0700 1 --2- 172.21.15.29:0/2877374839 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fded8102e10 0x7fded8113b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:17:19.025 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:17.196+0000 7fdedd0af700 1 -- 172.21.15.29:0/2877374839 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fded8102e10 msgr2=0x7fded8113b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:17:19.025 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:17.196+0000 7fdedd0af700 1 --2- 172.21.15.29:0/2877374839 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fded8102e10 0x7fded8113b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:17:19.025 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:17.196+0000 7fdedd0af700 1 -- 172.21.15.29:0/2877374839 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdec8005040 con 0x7fded81037b0 2024-03-18T01:17:19.025 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:17.196+0000 7fdedd0af700 1 --2- 172.21.15.29:0/2877374839 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fded81037b0 0x7fded81140c0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fdec800b3b0 tx=0x7fdec8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:17:19.025 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:17.196+0000 7fdece7fc700 1 -- 172.21.15.29:0/2877374839 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdec800e040 con 0x7fded81037b0 2024-03-18T01:17:19.025 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:17.196+0000 7fdece7fc700 1 -- 172.21.15.29:0/2877374839 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fdec8009b10 con 0x7fded81037b0 2024-03-18T01:17:19.025 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:17.196+0000 7fdedfb14700 1 -- 172.21.15.29:0/2877374839 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fded810dda0 con 0x7fded81037b0 2024-03-18T01:17:19.025 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:17.196+0000 7fdece7fc700 1 -- 172.21.15.29:0/2877374839 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdec8013420 con 0x7fded81037b0 2024-03-18T01:17:19.026 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:17.196+0000 7fdedfb14700 1 -- 172.21.15.29:0/2877374839 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fded810e360 con 0x7fded81037b0 2024-03-18T01:17:19.026 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:17.197+0000 7fdedfb14700 1 -- 172.21.15.29:0/2877374839 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fded810e840 con 0x7fded81037b0 2024-03-18T01:17:19.026 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:17.197+0000 7fdece7fc700 1 -- 172.21.15.29:0/2877374839 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7fdec801c7c0 con 0x7fded81037b0 2024-03-18T01:17:19.026 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:17.198+0000 7fdece7fc700 1 --2- 172.21.15.29:0/2877374839 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7fdec40692a0 0x7fdec406b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:17:19.026 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:17.198+0000 7fdedd8b0700 1 --2- 172.21.15.29:0/2877374839 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7fdec40692a0 0x7fdec406b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:17:19.026 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:17.198+0000 7fdece7fc700 1 -- 172.21.15.29:0/2877374839 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fdec8087970 con 0x7fded81037b0 2024-03-18T01:17:19.026 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:17.198+0000 7fdece7fc700 1 --2- 172.21.15.29:0/2877374839 >> [v2:172.21.15.29:6802/895619893,v1:172.21.15.29:6803/895619893] conn(0x7fdec406ef70 0x7fdec4071380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:17:19.026 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:17.198+0000 7fdece7fc700 1 -- 172.21.15.29:0/2877374839 --> [v2:172.21.15.29:6802/895619893,v1:172.21.15.29:6803/895619893] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fdec4071a30 con 0x7fdec406ef70 2024-03-18T01:17:19.026 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:17.198+0000 7fdece7fc700 1 -- 172.21.15.29:0/2877374839 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7fdec808c220 con 0x7fded81037b0 2024-03-18T01:17:19.027 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:17.198+0000 7fdede0b1700 1 --2- 172.21.15.29:0/2877374839 >> [v2:172.21.15.29:6802/895619893,v1:172.21.15.29:6803/895619893] conn(0x7fdec406ef70 0x7fdec4071380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:17:19.027 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:17.198+0000 7fdedd8b0700 1 --2- 172.21.15.29:0/2877374839 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7fdec40692a0 0x7fdec406b750 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fded4004f60 tx=0x7fded400b040 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:17:19.027 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:17.198+0000 7fdede0b1700 1 --2- 172.21.15.29:0/2877374839 >> [v2:172.21.15.29:6802/895619893,v1:172.21.15.29:6803/895619893] conn(0x7fdec406ef70 0x7fdec4071380 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:17:19.027 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:17.199+0000 7fdece7fc700 1 -- 172.21.15.29:0/2877374839 <== osd.0 v2:172.21.15.29:6802/895619893 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7fdec4071a30 con 0x7fdec406ef70 2024-03-18T01:17:19.027 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:17.214+0000 7fdedfb14700 1 -- 172.21.15.29:0/2877374839 --> [v2:172.21.15.29:6802/895619893,v1:172.21.15.29:6803/895619893] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fded805e6c0 con 0x7fdec406ef70 2024-03-18T01:17:19.027 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:17.214+0000 7fdece7fc700 1 -- 172.21.15.29:0/2877374839 <== osd.0 v2:172.21.15.29:6802/895619893 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fded805e6c0 con 0x7fdec406ef70 2024-03-18T01:17:19.027 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:17.214+0000 7fdedfb14700 1 -- 172.21.15.29:0/2877374839 >> [v2:172.21.15.29:6802/895619893,v1:172.21.15.29:6803/895619893] conn(0x7fdec406ef70 msgr2=0x7fdec4071380 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:17:19.028 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:17.214+0000 7fdedfb14700 1 --2- 172.21.15.29:0/2877374839 >> [v2:172.21.15.29:6802/895619893,v1:172.21.15.29:6803/895619893] conn(0x7fdec406ef70 0x7fdec4071380 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:17:19.028 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:17.215+0000 7fdedfb14700 1 -- 172.21.15.29:0/2877374839 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7fdec40692a0 msgr2=0x7fdec406b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:17:19.028 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:17.215+0000 7fdedfb14700 1 --2- 172.21.15.29:0/2877374839 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7fdec40692a0 0x7fdec406b750 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fded4004f60 tx=0x7fded400b040 comp rx=0 tx=0).stop 2024-03-18T01:17:19.028 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:17.215+0000 7fdedfb14700 1 -- 172.21.15.29:0/2877374839 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fded81037b0 msgr2=0x7fded81140c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:17:19.028 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:17.215+0000 7fdedfb14700 1 --2- 172.21.15.29:0/2877374839 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fded81037b0 0x7fded81140c0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fdec800b3b0 tx=0x7fdec8009e30 comp rx=0 tx=0).stop 2024-03-18T01:17:19.028 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:17.215+0000 7fdedfb14700 1 -- 172.21.15.29:0/2877374839 shutdown_connections 2024-03-18T01:17:19.028 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:17.215+0000 7fdedfb14700 1 --2- 172.21.15.29:0/2877374839 >> [v2:172.21.15.29:6802/895619893,v1:172.21.15.29:6803/895619893] conn(0x7fdec406ef70 0x7fdec4071380 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:17:19.028 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:17.215+0000 7fdedfb14700 1 --2- 172.21.15.29:0/2877374839 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7fdec40692a0 0x7fdec406b750 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:17:19.028 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:17.215+0000 7fdedfb14700 1 --2- 172.21.15.29:0/2877374839 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fded8102e10 0x7fded8113b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:17:19.028 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:17.215+0000 7fdedfb14700 1 --2- 172.21.15.29:0/2877374839 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fded81037b0 0x7fded81140c0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:17:19.028 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:17.215+0000 7fdedfb14700 1 -- 172.21.15.29:0/2877374839 >> 172.21.15.29:0/2877374839 conn(0x7fded80f89a0 msgr2=0x7fded80face0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:17:19.028 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:17.215+0000 7fdedfb14700 1 -- 172.21.15.29:0/2877374839 shutdown_connections 2024-03-18T01:17:19.028 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:17.215+0000 7fdedfb14700 1 -- 172.21.15.29:0/2877374839 wait complete. 2024-03-18T01:17:19.768 INFO:teuthology.orchestra.run.smithi029.stdout:38654705700 2024-03-18T01:17:19.768 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 37f51182-e4c4-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.0 2024-03-18T01:17:20.206 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:17:20 smithi029 ceph-mon[29331]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-18T01:17:20.457 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:20.456+0000 7fa3df71a700 1 -- 172.21.15.29:0/483188823 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa3d80fe9d0 msgr2=0x7fa3d80fee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:17:20.457 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:20.456+0000 7fa3df71a700 1 --2- 172.21.15.29:0/483188823 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa3d80fe9d0 0x7fa3d80fee40 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fa3c8004940 tx=0x7fa3c8009e30 comp rx=0 tx=0).stop 2024-03-18T01:17:20.457 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:20.456+0000 7fa3df71a700 1 -- 172.21.15.29:0/483188823 shutdown_connections 2024-03-18T01:17:20.457 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:20.456+0000 7fa3df71a700 1 --2- 172.21.15.29:0/483188823 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa3d80fe9d0 0x7fa3d80fee40 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:17:20.457 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:20.456+0000 7fa3df71a700 1 --2- 172.21.15.29:0/483188823 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa3d80fd1d0 0x7fa3d80fd5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:17:20.458 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:20.457+0000 7fa3df71a700 1 -- 172.21.15.29:0/483188823 >> 172.21.15.29:0/483188823 conn(0x7fa3d80f89b0 msgr2=0x7fa3d80fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:17:20.458 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:20.457+0000 7fa3df71a700 1 -- 172.21.15.29:0/483188823 shutdown_connections 2024-03-18T01:17:20.458 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:20.457+0000 7fa3df71a700 1 -- 172.21.15.29:0/483188823 wait complete. 2024-03-18T01:17:20.459 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:20.457+0000 7fa3df71a700 1 Processor -- start 2024-03-18T01:17:20.459 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:20.457+0000 7fa3df71a700 1 -- start start 2024-03-18T01:17:20.459 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:20.458+0000 7fa3df71a700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa3d80fd1d0 0x7fa3d8113a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:17:20.459 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:20.458+0000 7fa3df71a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa3d80fe9d0 0x7fa3d8113f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:17:20.459 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:20.458+0000 7fa3df71a700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3d8114650 con 0x7fa3d80fe9d0 2024-03-18T01:17:20.460 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:20.458+0000 7fa3df71a700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3d810dba0 con 0x7fa3d80fd1d0 2024-03-18T01:17:20.460 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:20.458+0000 7fa3dccb5700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa3d80fe9d0 0x7fa3d8113f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:17:20.460 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:20.458+0000 7fa3dccb5700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa3d80fe9d0 0x7fa3d8113f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60386/0 (socket says 172.21.15.29:60386) 2024-03-18T01:17:20.460 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:20.458+0000 7fa3dccb5700 1 -- 172.21.15.29:0/3137655492 learned_addr learned my addr 172.21.15.29:0/3137655492 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-18T01:17:20.460 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:20.458+0000 7fa3dd4b6700 1 --2- 172.21.15.29:0/3137655492 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa3d80fd1d0 0x7fa3d8113a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:17:20.460 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:20.458+0000 7fa3dccb5700 1 -- 172.21.15.29:0/3137655492 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa3d80fd1d0 msgr2=0x7fa3d8113a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:17:20.460 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:20.458+0000 7fa3dccb5700 1 --2- 172.21.15.29:0/3137655492 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa3d80fd1d0 0x7fa3d8113a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:17:20.460 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:20.458+0000 7fa3dccb5700 1 -- 172.21.15.29:0/3137655492 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3c8005040 con 0x7fa3d80fe9d0 2024-03-18T01:17:20.460 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:20.458+0000 7fa3dccb5700 1 --2- 172.21.15.29:0/3137655492 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa3d80fe9d0 0x7fa3d8113f70 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fa3c800b3b0 tx=0x7fa3c8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:17:20.460 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:20.459+0000 7fa3cdffb700 1 -- 172.21.15.29:0/3137655492 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa3c800e040 con 0x7fa3d80fe9d0 2024-03-18T01:17:20.460 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:20.459+0000 7fa3cdffb700 1 -- 172.21.15.29:0/3137655492 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa3c8013420 con 0x7fa3d80fe9d0 2024-03-18T01:17:20.460 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:20.459+0000 7fa3cdffb700 1 -- 172.21.15.29:0/3137655492 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa3c801c630 con 0x7fa3d80fe9d0 2024-03-18T01:17:20.460 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:20.459+0000 7fa3df71a700 1 -- 172.21.15.29:0/3137655492 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3d810ddc0 con 0x7fa3d80fe9d0 2024-03-18T01:17:20.461 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:20.459+0000 7fa3df71a700 1 -- 172.21.15.29:0/3137655492 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3d810e300 con 0x7fa3d80fe9d0 2024-03-18T01:17:20.461 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:20.460+0000 7fa3df71a700 1 -- 172.21.15.29:0/3137655492 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fa3d810e830 con 0x7fa3d80fe9d0 2024-03-18T01:17:20.461 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:20.461+0000 7fa3cdffb700 1 -- 172.21.15.29:0/3137655492 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7fa3c8013590 con 0x7fa3d80fe9d0 2024-03-18T01:17:20.462 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:20.461+0000 7fa3cdffb700 1 --2- 172.21.15.29:0/3137655492 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7fa3c40692a0 0x7fa3c406b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:17:20.462 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:20.461+0000 7fa3cdffb700 1 -- 172.21.15.29:0/3137655492 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fa3c8087f90 con 0x7fa3d80fe9d0 2024-03-18T01:17:20.462 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:20.461+0000 7fa3dd4b6700 1 --2- 172.21.15.29:0/3137655492 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7fa3c40692a0 0x7fa3c406b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:17:20.462 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:20.462+0000 7fa3cdffb700 1 --2- 172.21.15.29:0/3137655492 >> [v2:172.21.15.90:6808/2675255441,v1:172.21.15.90:6809/2675255441] conn(0x7fa3c406ef70 0x7fa3c4071380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:17:20.462 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:20.462+0000 7fa3cdffb700 1 -- 172.21.15.29:0/3137655492 --> [v2:172.21.15.90:6808/2675255441,v1:172.21.15.90:6809/2675255441] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa3c4071a30 con 0x7fa3c406ef70 2024-03-18T01:17:20.462 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:20.462+0000 7fa3cdffb700 1 -- 172.21.15.29:0/3137655492 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa3c808a600 con 0x7fa3d80fe9d0 2024-03-18T01:17:20.462 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:20.462+0000 7fa3ddcb7700 1 --2- 172.21.15.29:0/3137655492 >> [v2:172.21.15.90:6808/2675255441,v1:172.21.15.90:6809/2675255441] conn(0x7fa3c406ef70 0x7fa3c4071380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:17:20.462 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:20.462+0000 7fa3dd4b6700 1 --2- 172.21.15.29:0/3137655492 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7fa3c40692a0 0x7fa3c406b750 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fa3d4009bf0 tx=0x7fa3d40091a0 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:17:20.463 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:20.462+0000 7fa3ddcb7700 1 --2- 172.21.15.29:0/3137655492 >> [v2:172.21.15.90:6808/2675255441,v1:172.21.15.90:6809/2675255441] conn(0x7fa3c406ef70 0x7fa3c4071380 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:17:20.463 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:20.463+0000 7fa3cdffb700 1 -- 172.21.15.29:0/3137655492 <== osd.4 v2:172.21.15.90:6808/2675255441 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7fa3c4071a30 con 0x7fa3c406ef70 2024-03-18T01:17:20.478 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:20.478+0000 7fa3df71a700 1 -- 172.21.15.29:0/3137655492 --> [v2:172.21.15.90:6808/2675255441,v1:172.21.15.90:6809/2675255441] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fa3d80f9710 con 0x7fa3c406ef70 2024-03-18T01:17:20.479 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:20.478+0000 7fa3cdffb700 1 -- 172.21.15.29:0/3137655492 <== osd.4 v2:172.21.15.90:6808/2675255441 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7fa3d80f9710 con 0x7fa3c406ef70 2024-03-18T01:17:20.480 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:20.479+0000 7fa3df71a700 1 -- 172.21.15.29:0/3137655492 >> [v2:172.21.15.90:6808/2675255441,v1:172.21.15.90:6809/2675255441] conn(0x7fa3c406ef70 msgr2=0x7fa3c4071380 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:17:20.480 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:20.479+0000 7fa3df71a700 1 --2- 172.21.15.29:0/3137655492 >> [v2:172.21.15.90:6808/2675255441,v1:172.21.15.90:6809/2675255441] conn(0x7fa3c406ef70 0x7fa3c4071380 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:17:20.480 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:20.479+0000 7fa3df71a700 1 -- 172.21.15.29:0/3137655492 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7fa3c40692a0 msgr2=0x7fa3c406b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:17:20.480 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:20.479+0000 7fa3df71a700 1 --2- 172.21.15.29:0/3137655492 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7fa3c40692a0 0x7fa3c406b750 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fa3d4009bf0 tx=0x7fa3d40091a0 comp rx=0 tx=0).stop 2024-03-18T01:17:20.480 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:20.479+0000 7fa3df71a700 1 -- 172.21.15.29:0/3137655492 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa3d80fe9d0 msgr2=0x7fa3d8113f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:17:20.480 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:20.479+0000 7fa3df71a700 1 --2- 172.21.15.29:0/3137655492 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa3d80fe9d0 0x7fa3d8113f70 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fa3c800b3b0 tx=0x7fa3c8009e30 comp rx=0 tx=0).stop 2024-03-18T01:17:20.480 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:20.479+0000 7fa3df71a700 1 -- 172.21.15.29:0/3137655492 shutdown_connections 2024-03-18T01:17:20.480 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:20.479+0000 7fa3df71a700 1 --2- 172.21.15.29:0/3137655492 >> [v2:172.21.15.90:6808/2675255441,v1:172.21.15.90:6809/2675255441] conn(0x7fa3c406ef70 0x7fa3c4071380 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:17:20.480 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:20.479+0000 7fa3df71a700 1 --2- 172.21.15.29:0/3137655492 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7fa3c40692a0 0x7fa3c406b750 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:17:20.480 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:20.479+0000 7fa3df71a700 1 --2- 172.21.15.29:0/3137655492 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa3d80fd1d0 0x7fa3d8113a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:17:20.480 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:20.479+0000 7fa3df71a700 1 --2- 172.21.15.29:0/3137655492 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa3d80fe9d0 0x7fa3d8113f70 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:17:20.481 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:20.479+0000 7fa3df71a700 1 -- 172.21.15.29:0/3137655492 >> 172.21.15.29:0/3137655492 conn(0x7fa3d80f89b0 msgr2=0x7fa3d80f8d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:17:20.481 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:20.479+0000 7fa3df71a700 1 -- 172.21.15.29:0/3137655492 shutdown_connections 2024-03-18T01:17:20.481 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:20.479+0000 7fa3df71a700 1 -- 172.21.15.29:0/3137655492 wait complete. 2024-03-18T01:17:20.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:17:20 smithi090 ceph-mon[36989]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-18T01:17:20.661 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:20.660+0000 7fd3ba2d1700 1 -- 172.21.15.29:0/1176072756 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd3b4101d20 msgr2=0x7fd3b41020f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:17:20.662 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:20.660+0000 7fd3ba2d1700 1 --2- 172.21.15.29:0/1176072756 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd3b4101d20 0x7fd3b41020f0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fd3a4004940 tx=0x7fd3a4009e30 comp rx=0 tx=0).stop 2024-03-18T01:17:20.662 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:20.660+0000 7fd3ba2d1700 1 -- 172.21.15.29:0/1176072756 shutdown_connections 2024-03-18T01:17:20.662 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:20.660+0000 7fd3ba2d1700 1 --2- 172.21.15.29:0/1176072756 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd3b41026c0 0x7fd3b40fbd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:17:20.662 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:20.660+0000 7fd3ba2d1700 1 --2- 172.21.15.29:0/1176072756 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd3b4101d20 0x7fd3b41020f0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:17:20.662 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:20.660+0000 7fd3ba2d1700 1 -- 172.21.15.29:0/1176072756 >> 172.21.15.29:0/1176072756 conn(0x7fd3b40f7870 msgr2=0x7fd3b40f9c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:17:20.662 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:20.660+0000 7fd3ba2d1700 1 -- 172.21.15.29:0/1176072756 shutdown_connections 2024-03-18T01:17:20.662 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:20.660+0000 7fd3ba2d1700 1 -- 172.21.15.29:0/1176072756 wait complete. 2024-03-18T01:17:20.662 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:20.661+0000 7fd3ba2d1700 1 Processor -- start 2024-03-18T01:17:20.662 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:20.661+0000 7fd3ba2d1700 1 -- start start 2024-03-18T01:17:20.662 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:20.661+0000 7fd3ba2d1700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd3b4101d20 0x7fd3b4113970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:17:20.662 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:20.661+0000 7fd3ba2d1700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd3b41026c0 0x7fd3b4113eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:17:20.662 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:20.661+0000 7fd3ba2d1700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3b4114590 con 0x7fd3b4101d20 2024-03-18T01:17:20.663 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:20.661+0000 7fd3ba2d1700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3b410db00 con 0x7fd3b41026c0 2024-03-18T01:17:20.663 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:20.661+0000 7fd3b3fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd3b4101d20 0x7fd3b4113970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:17:20.663 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:20.661+0000 7fd3b3fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd3b4101d20 0x7fd3b4113970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60404/0 (socket says 172.21.15.29:60404) 2024-03-18T01:17:20.663 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:20.661+0000 7fd3b3fff700 1 -- 172.21.15.29:0/3029073261 learned_addr learned my addr 172.21.15.29:0/3029073261 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-18T01:17:20.665 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:20.662+0000 7fd3b3fff700 1 -- 172.21.15.29:0/3029073261 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd3b41026c0 msgr2=0x7fd3b4113eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:17:20.665 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:20.662+0000 7fd3b37fe700 1 --2- 172.21.15.29:0/3029073261 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd3b41026c0 0x7fd3b4113eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:17:20.665 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:20.662+0000 7fd3b3fff700 1 --2- 172.21.15.29:0/3029073261 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd3b41026c0 0x7fd3b4113eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:17:20.665 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:20.662+0000 7fd3b3fff700 1 -- 172.21.15.29:0/3029073261 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3a4005040 con 0x7fd3b4101d20 2024-03-18T01:17:20.665 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:20.662+0000 7fd3b37fe700 1 --2- 172.21.15.29:0/3029073261 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd3b41026c0 0x7fd3b4113eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T01:17:20.665 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:20.662+0000 7fd3b3fff700 1 --2- 172.21.15.29:0/3029073261 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd3b4101d20 0x7fd3b4113970 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fd3a400b3b0 tx=0x7fd3a4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:17:20.665 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:20.662+0000 7fd3b0ff9700 1 -- 172.21.15.29:0/3029073261 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd3a400e040 con 0x7fd3b4101d20 2024-03-18T01:17:20.665 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:20.662+0000 7fd3b0ff9700 1 -- 172.21.15.29:0/3029073261 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd3a400bbc0 con 0x7fd3b4101d20 2024-03-18T01:17:20.665 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:20.662+0000 7fd3ba2d1700 1 -- 172.21.15.29:0/3029073261 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3b410dd80 con 0x7fd3b4101d20 2024-03-18T01:17:20.665 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:20.662+0000 7fd3b0ff9700 1 -- 172.21.15.29:0/3029073261 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd3a401c3f0 con 0x7fd3b4101d20 2024-03-18T01:17:20.665 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:20.662+0000 7fd3ba2d1700 1 -- 172.21.15.29:0/3029073261 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3b410e340 con 0x7fd3b4101d20 2024-03-18T01:17:20.665 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:20.663+0000 7fd3ba2d1700 1 -- 172.21.15.29:0/3029073261 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fd3b410e860 con 0x7fd3b4101d20 2024-03-18T01:17:20.666 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:20.663+0000 7fd3b0ff9700 1 -- 172.21.15.29:0/3029073261 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7fd3a400bd30 con 0x7fd3b4101d20 2024-03-18T01:17:20.666 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:20.664+0000 7fd3b0ff9700 1 --2- 172.21.15.29:0/3029073261 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7fd39c0692a0 0x7fd39c06b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:17:20.666 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:20.664+0000 7fd3b0ff9700 1 -- 172.21.15.29:0/3029073261 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fd3a4088b60 con 0x7fd3b4101d20 2024-03-18T01:17:20.666 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:20.664+0000 7fd3b37fe700 1 --2- 172.21.15.29:0/3029073261 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7fd39c0692a0 0x7fd39c06b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:17:20.666 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:20.664+0000 7fd3b0ff9700 1 --2- 172.21.15.29:0/3029073261 >> [v2:172.21.15.90:6816/2630668390,v1:172.21.15.90:6817/2630668390] conn(0x7fd39c06ef70 0x7fd39c071380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:17:20.666 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:20.664+0000 7fd3b0ff9700 1 -- 172.21.15.29:0/3029073261 --> [v2:172.21.15.90:6816/2630668390,v1:172.21.15.90:6817/2630668390] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd39c071a30 con 0x7fd39c06ef70 2024-03-18T01:17:20.666 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:20.664+0000 7fd3b0ff9700 1 -- 172.21.15.29:0/3029073261 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd3a4088f40 con 0x7fd3b4101d20 2024-03-18T01:17:20.666 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:20.664+0000 7fd3b886e700 1 --2- 172.21.15.29:0/3029073261 >> [v2:172.21.15.90:6816/2630668390,v1:172.21.15.90:6817/2630668390] conn(0x7fd39c06ef70 0x7fd39c071380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:17:20.666 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:20.664+0000 7fd3b37fe700 1 --2- 172.21.15.29:0/3029073261 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7fd39c0692a0 0x7fd39c06b750 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fd3a8006fd0 tx=0x7fd3a8008040 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:17:20.666 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:20.665+0000 7fd3b886e700 1 --2- 172.21.15.29:0/3029073261 >> [v2:172.21.15.90:6816/2630668390,v1:172.21.15.90:6817/2630668390] conn(0x7fd39c06ef70 0x7fd39c071380 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:17:20.667 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:20.665+0000 7fd3b0ff9700 1 -- 172.21.15.29:0/3029073261 <== osd.5 v2:172.21.15.90:6816/2630668390 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7fd39c071a30 con 0x7fd39c06ef70 2024-03-18T01:17:20.680 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:20.680+0000 7fd3ba2d1700 1 -- 172.21.15.29:0/3029073261 --> [v2:172.21.15.90:6816/2630668390,v1:172.21.15.90:6817/2630668390] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fd3b405e6c0 con 0x7fd39c06ef70 2024-03-18T01:17:20.681 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:20.681+0000 7fd3b0ff9700 1 -- 172.21.15.29:0/3029073261 <== osd.5 v2:172.21.15.90:6816/2630668390 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7fd3b405e6c0 con 0x7fd39c06ef70 2024-03-18T01:17:20.682 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:20.681+0000 7fd3ba2d1700 1 -- 172.21.15.29:0/3029073261 >> [v2:172.21.15.90:6816/2630668390,v1:172.21.15.90:6817/2630668390] conn(0x7fd39c06ef70 msgr2=0x7fd39c071380 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:17:20.682 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:20.681+0000 7fd3ba2d1700 1 --2- 172.21.15.29:0/3029073261 >> [v2:172.21.15.90:6816/2630668390,v1:172.21.15.90:6817/2630668390] conn(0x7fd39c06ef70 0x7fd39c071380 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:17:20.682 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:20.681+0000 7fd3ba2d1700 1 -- 172.21.15.29:0/3029073261 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7fd39c0692a0 msgr2=0x7fd39c06b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:17:20.682 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:20.681+0000 7fd3ba2d1700 1 --2- 172.21.15.29:0/3029073261 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7fd39c0692a0 0x7fd39c06b750 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fd3a8006fd0 tx=0x7fd3a8008040 comp rx=0 tx=0).stop 2024-03-18T01:17:20.682 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:20.681+0000 7fd3ba2d1700 1 -- 172.21.15.29:0/3029073261 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd3b4101d20 msgr2=0x7fd3b4113970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:17:20.682 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:20.681+0000 7fd3ba2d1700 1 --2- 172.21.15.29:0/3029073261 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd3b4101d20 0x7fd3b4113970 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fd3a400b3b0 tx=0x7fd3a4009e30 comp rx=0 tx=0).stop 2024-03-18T01:17:20.682 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:20.681+0000 7fd3ba2d1700 1 -- 172.21.15.29:0/3029073261 shutdown_connections 2024-03-18T01:17:20.682 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:20.681+0000 7fd3ba2d1700 1 --2- 172.21.15.29:0/3029073261 >> [v2:172.21.15.90:6816/2630668390,v1:172.21.15.90:6817/2630668390] conn(0x7fd39c06ef70 0x7fd39c071380 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:17:20.683 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:20.681+0000 7fd3ba2d1700 1 --2- 172.21.15.29:0/3029073261 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7fd39c0692a0 0x7fd39c06b750 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:17:20.683 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:20.681+0000 7fd3ba2d1700 1 --2- 172.21.15.29:0/3029073261 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd3b4101d20 0x7fd3b4113970 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:17:20.683 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:20.681+0000 7fd3ba2d1700 1 --2- 172.21.15.29:0/3029073261 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd3b41026c0 0x7fd3b4113eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:17:20.683 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:20.681+0000 7fd3ba2d1700 1 -- 172.21.15.29:0/3029073261 >> 172.21.15.29:0/3029073261 conn(0x7fd3b40f7870 msgr2=0x7fd3b40f9870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:17:20.683 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:20.681+0000 7fd3ba2d1700 1 -- 172.21.15.29:0/3029073261 shutdown_connections 2024-03-18T01:17:20.683 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:20.681+0000 7fd3ba2d1700 1 -- 172.21.15.29:0/3029073261 wait complete. 2024-03-18T01:17:21.839 INFO:teuthology.orchestra.run.smithi029.stdout:146028888080 2024-03-18T01:17:21.839 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 37f51182-e4c4-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.4 2024-03-18T01:17:22.340 INFO:teuthology.orchestra.run.smithi029.stdout:60129542174 2024-03-18T01:17:22.340 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 37f51182-e4c4-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.1 2024-03-18T01:17:22.492 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:17:22 smithi029 ceph-mon[29331]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-18T01:17:22.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:17:22 smithi090 ceph-mon[36989]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-18T01:17:22.990 INFO:teuthology.orchestra.run.smithi029.stdout:167503724555 2024-03-18T01:17:22.990 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 37f51182-e4c4-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.5 2024-03-18T01:17:23.175 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:23.172+0000 7fbb7f9b1700 1 -- 172.21.15.29:0/291859247 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbb78101d10 msgr2=0x7fbb781020e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:17:23.175 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:23.172+0000 7fbb7f9b1700 1 --2- 172.21.15.29:0/291859247 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbb78101d10 0x7fbb781020e0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fbb70009ce0 tx=0x7fbb70004d10 comp rx=0 tx=0).stop 2024-03-18T01:17:23.175 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:23.172+0000 7fbb7f9b1700 1 -- 172.21.15.29:0/291859247 shutdown_connections 2024-03-18T01:17:23.175 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:23.172+0000 7fbb7f9b1700 1 --2- 172.21.15.29:0/291859247 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbb781026b0 0x7fbb780fbd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:17:23.175 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:23.172+0000 7fbb7f9b1700 1 --2- 172.21.15.29:0/291859247 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbb78101d10 0x7fbb781020e0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:17:23.176 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:23.172+0000 7fbb7f9b1700 1 -- 172.21.15.29:0/291859247 >> 172.21.15.29:0/291859247 conn(0x7fbb780f7880 msgr2=0x7fbb780f9c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:17:23.176 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:23.173+0000 7fbb7f9b1700 1 -- 172.21.15.29:0/291859247 shutdown_connections 2024-03-18T01:17:23.176 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:23.173+0000 7fbb7f9b1700 1 -- 172.21.15.29:0/291859247 wait complete. 2024-03-18T01:17:23.176 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:23.173+0000 7fbb7f9b1700 1 Processor -- start 2024-03-18T01:17:23.176 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:23.173+0000 7fbb7f9b1700 1 -- start start 2024-03-18T01:17:23.176 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:23.174+0000 7fbb7f9b1700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbb78101d10 0x7fbb7810f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:17:23.176 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:23.174+0000 7fbb7f9b1700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbb781026b0 0x7fbb7810f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:17:23.176 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:23.174+0000 7fbb7f9b1700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb7810fc20 con 0x7fbb78101d10 2024-03-18T01:17:23.176 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:23.174+0000 7fbb7f9b1700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb7810fd90 con 0x7fbb781026b0 2024-03-18T01:17:23.176 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:23.174+0000 7fbb7d74d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbb78101d10 0x7fbb7810f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:17:23.176 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:23.174+0000 7fbb7d74d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbb78101d10 0x7fbb7810f1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51856/0 (socket says 172.21.15.29:51856) 2024-03-18T01:17:23.176 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:23.174+0000 7fbb7d74d700 1 -- 172.21.15.29:0/1737274680 learned_addr learned my addr 172.21.15.29:0/1737274680 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-18T01:17:23.177 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:23.174+0000 7fbb7cf4c700 1 --2- 172.21.15.29:0/1737274680 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbb781026b0 0x7fbb7810f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:17:23.177 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:23.174+0000 7fbb7d74d700 1 -- 172.21.15.29:0/1737274680 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbb781026b0 msgr2=0x7fbb7810f6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:17:23.177 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:23.174+0000 7fbb7d74d700 1 --2- 172.21.15.29:0/1737274680 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbb781026b0 0x7fbb7810f6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:17:23.177 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:23.174+0000 7fbb7d74d700 1 -- 172.21.15.29:0/1737274680 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb70009940 con 0x7fbb78101d10 2024-03-18T01:17:23.177 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:23.174+0000 7fbb7cf4c700 1 --2- 172.21.15.29:0/1737274680 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbb781026b0 0x7fbb7810f6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T01:17:23.177 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:23.174+0000 7fbb7d74d700 1 --2- 172.21.15.29:0/1737274680 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbb78101d10 0x7fbb7810f1a0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fbb7000be90 tx=0x7fbb70004d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:17:23.177 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:23.174+0000 7fbb75ffb700 1 -- 172.21.15.29:0/1737274680 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbb7000e040 con 0x7fbb78101d10 2024-03-18T01:17:23.177 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:23.174+0000 7fbb75ffb700 1 -- 172.21.15.29:0/1737274680 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fbb70008ea0 con 0x7fbb78101d10 2024-03-18T01:17:23.177 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:23.174+0000 7fbb7f9b1700 1 -- 172.21.15.29:0/1737274680 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb78118050 con 0x7fbb78101d10 2024-03-18T01:17:23.177 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:23.174+0000 7fbb75ffb700 1 -- 172.21.15.29:0/1737274680 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbb7001c810 con 0x7fbb78101d10 2024-03-18T01:17:23.177 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:23.175+0000 7fbb7f9b1700 1 -- 172.21.15.29:0/1737274680 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb78118560 con 0x7fbb78101d10 2024-03-18T01:17:23.178 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:23.176+0000 7fbb7f9b1700 1 -- 172.21.15.29:0/1737274680 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb780fd460 con 0x7fbb78101d10 2024-03-18T01:17:23.178 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:23.176+0000 7fbb75ffb700 1 -- 172.21.15.29:0/1737274680 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7fbb70012c00 con 0x7fbb78101d10 2024-03-18T01:17:23.178 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:23.177+0000 7fbb75ffb700 1 --2- 172.21.15.29:0/1737274680 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7fbb600692a0 0x7fbb6006b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:17:23.179 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:23.177+0000 7fbb75ffb700 1 -- 172.21.15.29:0/1737274680 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fbb70088ad0 con 0x7fbb78101d10 2024-03-18T01:17:23.179 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:23.177+0000 7fbb7cf4c700 1 --2- 172.21.15.29:0/1737274680 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7fbb600692a0 0x7fbb6006b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:17:23.179 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:23.178+0000 7fbb7cf4c700 1 --2- 172.21.15.29:0/1737274680 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7fbb600692a0 0x7fbb6006b750 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fbb6c007fd0 tx=0x7fbb6c009040 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:17:23.182 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:23.182+0000 7fbb75ffb700 1 -- 172.21.15.29:0/1737274680 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175520 (secure 0 0 0) 0x7fbb700550c0 con 0x7fbb78101d10 2024-03-18T01:17:23.387 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:23.386+0000 7fbb7f9b1700 1 -- 172.21.15.29:0/1737274680 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7fbb7805f5f0 con 0x7fbb78101d10 2024-03-18T01:17:23.387 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:23.386+0000 7fbb75ffb700 1 -- 172.21.15.29:0/1737274680 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fbb70004e10 con 0x7fbb78101d10 2024-03-18T01:17:23.387 INFO:teuthology.orchestra.run.smithi029.stdout:81604378650 2024-03-18T01:17:23.389 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:23.388+0000 7fbb7f9b1700 1 -- 172.21.15.29:0/1737274680 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7fbb600692a0 msgr2=0x7fbb6006b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:17:23.389 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:23.388+0000 7fbb7f9b1700 1 --2- 172.21.15.29:0/1737274680 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7fbb600692a0 0x7fbb6006b750 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fbb6c007fd0 tx=0x7fbb6c009040 comp rx=0 tx=0).stop 2024-03-18T01:17:23.389 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:23.388+0000 7fbb7f9b1700 1 -- 172.21.15.29:0/1737274680 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbb78101d10 msgr2=0x7fbb7810f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:17:23.389 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:23.388+0000 7fbb7f9b1700 1 --2- 172.21.15.29:0/1737274680 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbb78101d10 0x7fbb7810f1a0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fbb7000be90 tx=0x7fbb70004d10 comp rx=0 tx=0).stop 2024-03-18T01:17:23.389 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:23.388+0000 7fbb7f9b1700 1 -- 172.21.15.29:0/1737274680 shutdown_connections 2024-03-18T01:17:23.389 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:23.388+0000 7fbb7f9b1700 1 --2- 172.21.15.29:0/1737274680 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7fbb600692a0 0x7fbb6006b750 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:17:23.389 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:23.388+0000 7fbb7f9b1700 1 --2- 172.21.15.29:0/1737274680 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbb78101d10 0x7fbb7810f1a0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:17:23.389 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:23.388+0000 7fbb7f9b1700 1 --2- 172.21.15.29:0/1737274680 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbb781026b0 0x7fbb7810f6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:17:23.389 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:23.388+0000 7fbb7f9b1700 1 -- 172.21.15.29:0/1737274680 >> 172.21.15.29:0/1737274680 conn(0x7fbb780f7880 msgr2=0x7fbb780f88e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:17:23.389 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:23.389+0000 7fbb7f9b1700 1 -- 172.21.15.29:0/1737274680 shutdown_connections 2024-03-18T01:17:23.389 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:23.389+0000 7fbb7f9b1700 1 -- 172.21.15.29:0/1737274680 wait complete. 2024-03-18T01:17:24.283 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:17:24 smithi029 ceph-mon[29331]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-18T01:17:24.283 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:17:24 smithi029 ceph-mon[29331]: from='client.? 172.21.15.29:0/1737274680' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-18T01:17:24.309 INFO:tasks.cephadm.ceph_manager.ceph:need seq 81604378647 got 81604378650 for osd.2 2024-03-18T01:17:24.309 DEBUG:teuthology.parallel:result is None 2024-03-18T01:17:24.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:17:24 smithi090 ceph-mon[36989]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-18T01:17:24.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:17:24 smithi090 ceph-mon[36989]: from='client.? 172.21.15.29:0/1737274680' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-18T01:17:24.562 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/37f51182-e4c4-11ee-95c9-87774f69a715/mon.smithi029/config 2024-03-18T01:17:24.562 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/37f51182-e4c4-11ee-95c9-87774f69a715/mon.smithi029/config 2024-03-18T01:17:24.748 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/37f51182-e4c4-11ee-95c9-87774f69a715/mon.smithi029/config 2024-03-18T01:17:24.762 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:24.762+0000 7fadb9534700 1 -- 172.21.15.29:0/2995332626 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fadb4102e00 msgr2=0x7fadb41031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:17:24.763 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:24.762+0000 7fadb9534700 1 --2- 172.21.15.29:0/2995332626 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fadb4102e00 0x7fadb41031d0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fada4004940 tx=0x7fada4009e30 comp rx=0 tx=0).stop 2024-03-18T01:17:24.763 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:24.762+0000 7fadb9534700 1 -- 172.21.15.29:0/2995332626 shutdown_connections 2024-03-18T01:17:24.763 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:24.762+0000 7fadb9534700 1 --2- 172.21.15.29:0/2995332626 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fadb41037a0 0x7fadb40fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:17:24.763 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:24.762+0000 7fadb9534700 1 --2- 172.21.15.29:0/2995332626 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fadb4102e00 0x7fadb41031d0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:17:24.763 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:24.762+0000 7fadb9534700 1 -- 172.21.15.29:0/2995332626 >> 172.21.15.29:0/2995332626 conn(0x7fadb40f89b0 msgr2=0x7fadb40fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:17:24.763 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:24.762+0000 7fadb9534700 1 -- 172.21.15.29:0/2995332626 shutdown_connections 2024-03-18T01:17:24.763 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:24.762+0000 7fadb9534700 1 -- 172.21.15.29:0/2995332626 wait complete. 2024-03-18T01:17:24.764 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:24.763+0000 7fadb9534700 1 Processor -- start 2024-03-18T01:17:24.764 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:24.763+0000 7fadb9534700 1 -- start start 2024-03-18T01:17:24.765 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:24.763+0000 7fadb9534700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fadb4102e00 0x7fadb40702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:17:24.765 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:24.763+0000 7fadb9534700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fadb41037a0 0x7fadb4070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:17:24.765 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:24.763+0000 7fadb9534700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadb4070d90 con 0x7fadb41037a0 2024-03-18T01:17:24.765 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:24.763+0000 7fadb9534700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadb4070ed0 con 0x7fadb4102e00 2024-03-18T01:17:24.765 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:24.763+0000 7fadb27fc700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fadb41037a0 0x7fadb4070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:17:24.765 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:24.763+0000 7fadb27fc700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fadb41037a0 0x7fadb4070800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51876/0 (socket says 172.21.15.29:51876) 2024-03-18T01:17:24.765 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:24.763+0000 7fadb27fc700 1 -- 172.21.15.29:0/3566514446 learned_addr learned my addr 172.21.15.29:0/3566514446 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-18T01:17:24.765 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:24.764+0000 7fadb2ffd700 1 --2- 172.21.15.29:0/3566514446 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fadb4102e00 0x7fadb40702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:17:24.765 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:24.764+0000 7fadb27fc700 1 -- 172.21.15.29:0/3566514446 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fadb4102e00 msgr2=0x7fadb40702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:17:24.765 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:24.764+0000 7fadb27fc700 1 --2- 172.21.15.29:0/3566514446 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fadb4102e00 0x7fadb40702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:17:24.765 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:24.764+0000 7fadb27fc700 1 -- 172.21.15.29:0/3566514446 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fada4005040 con 0x7fadb41037a0 2024-03-18T01:17:24.765 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:24.764+0000 7fadb27fc700 1 --2- 172.21.15.29:0/3566514446 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fadb41037a0 0x7fadb4070800 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fada8002a50 tx=0x7fada8004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:17:24.765 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:24.764+0000 7fad9b7fe700 1 -- 172.21.15.29:0/3566514446 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fada8007cc0 con 0x7fadb41037a0 2024-03-18T01:17:24.765 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:24.764+0000 7fad9b7fe700 1 -- 172.21.15.29:0/3566514446 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fada800f040 con 0x7fadb41037a0 2024-03-18T01:17:24.766 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:24.764+0000 7fad9b7fe700 1 -- 172.21.15.29:0/3566514446 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fada800da40 con 0x7fadb41037a0 2024-03-18T01:17:24.766 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:24.764+0000 7fadb9534700 1 -- 172.21.15.29:0/3566514446 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fadb4119dd0 con 0x7fadb41037a0 2024-03-18T01:17:24.766 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:24.764+0000 7fadb9534700 1 -- 172.21.15.29:0/3566514446 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fadb411a250 con 0x7fadb41037a0 2024-03-18T01:17:24.766 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:24.766+0000 7fadb9534700 1 -- 172.21.15.29:0/3566514446 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fadb41052d0 con 0x7fadb41037a0 2024-03-18T01:17:24.766 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:24.766+0000 7fad9b7fe700 1 -- 172.21.15.29:0/3566514446 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7fada80097c0 con 0x7fadb41037a0 2024-03-18T01:17:24.767 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:24.766+0000 7fad9b7fe700 1 --2- 172.21.15.29:0/3566514446 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7fad9c069370 0x7fad9c06b820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:17:24.767 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:24.767+0000 7fad9b7fe700 1 -- 172.21.15.29:0/3566514446 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fada8086c10 con 0x7fadb41037a0 2024-03-18T01:17:24.767 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:24.767+0000 7fadb2ffd700 1 --2- 172.21.15.29:0/3566514446 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7fad9c069370 0x7fad9c06b820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:17:24.767 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:24.767+0000 7fadb2ffd700 1 --2- 172.21.15.29:0/3566514446 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7fad9c069370 0x7fad9c06b820 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fada4007f50 tx=0x7fada4009e30 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:17:24.774 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:24.774+0000 7fad9b7fe700 1 -- 172.21.15.29:0/3566514446 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175520 (secure 0 0 0) 0x7fada80531e0 con 0x7fadb41037a0 2024-03-18T01:17:24.791 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/37f51182-e4c4-11ee-95c9-87774f69a715/mon.smithi029/config 2024-03-18T01:17:24.977 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:24.977+0000 7fadb9534700 1 -- 172.21.15.29:0/3566514446 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7fadb4109f00 con 0x7fadb41037a0 2024-03-18T01:17:24.978 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:24.977+0000 7fad9b7fe700 1 -- 172.21.15.29:0/3566514446 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7fada8056800 con 0x7fadb41037a0 2024-03-18T01:17:24.978 INFO:teuthology.orchestra.run.smithi029.stdout:111669149716 2024-03-18T01:17:24.980 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:24.979+0000 7fadb9534700 1 -- 172.21.15.29:0/3566514446 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7fad9c069370 msgr2=0x7fad9c06b820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:17:24.980 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:24.979+0000 7fadb9534700 1 --2- 172.21.15.29:0/3566514446 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7fad9c069370 0x7fad9c06b820 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fada4007f50 tx=0x7fada4009e30 comp rx=0 tx=0).stop 2024-03-18T01:17:24.980 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:24.979+0000 7fadb9534700 1 -- 172.21.15.29:0/3566514446 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fadb41037a0 msgr2=0x7fadb4070800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:17:24.980 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:24.979+0000 7fadb9534700 1 --2- 172.21.15.29:0/3566514446 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fadb41037a0 0x7fadb4070800 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fada8002a50 tx=0x7fada8004600 comp rx=0 tx=0).stop 2024-03-18T01:17:24.980 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:24.980+0000 7fadb9534700 1 -- 172.21.15.29:0/3566514446 shutdown_connections 2024-03-18T01:17:24.980 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:24.980+0000 7fadb9534700 1 --2- 172.21.15.29:0/3566514446 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7fad9c069370 0x7fad9c06b820 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:17:24.980 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:24.980+0000 7fadb9534700 1 --2- 172.21.15.29:0/3566514446 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fadb4102e00 0x7fadb40702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:17:24.980 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:24.980+0000 7fadb9534700 1 --2- 172.21.15.29:0/3566514446 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fadb41037a0 0x7fadb4070800 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:17:24.980 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:24.980+0000 7fadb9534700 1 -- 172.21.15.29:0/3566514446 >> 172.21.15.29:0/3566514446 conn(0x7fadb40f89b0 msgr2=0x7fadb4102170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:17:24.981 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:24.980+0000 7fadb9534700 1 -- 172.21.15.29:0/3566514446 shutdown_connections 2024-03-18T01:17:24.981 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:24.980+0000 7fadb9534700 1 -- 172.21.15.29:0/3566514446 wait complete. 2024-03-18T01:17:25.472 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:17:25 smithi029 ceph-mon[29331]: from='client.? 172.21.15.29:0/3566514446' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-18T01:17:25.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:17:25 smithi090 ceph-mon[36989]: from='client.? 172.21.15.29:0/3566514446' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-18T01:17:26.319 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:17:26 smithi029 ceph-mon[29331]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-18T01:17:26.319 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:17:26 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-18T01:17:26.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:17:26 smithi090 ceph-mon[36989]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-18T01:17:26.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:17:26 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-18T01:17:27.783 INFO:tasks.cephadm.ceph_manager.ceph:need seq 111669149714 got 111669149716 for osd.3 2024-03-18T01:17:27.783 DEBUG:teuthology.parallel:result is None 2024-03-18T01:17:28.319 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:17:28 smithi029 ceph-mon[29331]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-18T01:17:28.525 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:17:28 smithi090 ceph-mon[36989]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-18T01:17:28.886 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:28.884+0000 7fb6083ef700 1 -- 172.21.15.29:0/1810938930 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb600102710 msgr2=0x7fb6000fbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:17:28.886 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:28.884+0000 7fb6083ef700 1 --2- 172.21.15.29:0/1810938930 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb600102710 0x7fb6000fbd80 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fb5f0004940 tx=0x7fb5f0009e30 comp rx=0 tx=0).stop 2024-03-18T01:17:28.886 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:28.885+0000 7fb6083ef700 1 -- 172.21.15.29:0/1810938930 shutdown_connections 2024-03-18T01:17:28.886 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:28.885+0000 7fb6083ef700 1 --2- 172.21.15.29:0/1810938930 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb600102710 0x7fb6000fbd80 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:17:28.886 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:28.885+0000 7fb6083ef700 1 --2- 172.21.15.29:0/1810938930 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb600101d70 0x7fb600102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:17:28.886 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:28.885+0000 7fb6083ef700 1 -- 172.21.15.29:0/1810938930 >> 172.21.15.29:0/1810938930 conn(0x7fb600072bc0 msgr2=0x7fb600072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:17:28.886 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:28.885+0000 7fb6083ef700 1 -- 172.21.15.29:0/1810938930 shutdown_connections 2024-03-18T01:17:28.886 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:28.885+0000 7fb6083ef700 1 -- 172.21.15.29:0/1810938930 wait complete. 2024-03-18T01:17:28.886 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:28.885+0000 7fb6083ef700 1 Processor -- start 2024-03-18T01:17:28.887 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:28.885+0000 7fb6083ef700 1 -- start start 2024-03-18T01:17:28.887 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:28.885+0000 7fb6083ef700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb600101d70 0x7fb60010c8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:17:28.887 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:28.885+0000 7fb6083ef700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb600102710 0x7fb60010ce10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:17:28.887 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:28.885+0000 7fb6083ef700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb60010d350 con 0x7fb600102710 2024-03-18T01:17:28.887 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:28.885+0000 7fb6083ef700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb60010d4c0 con 0x7fb600101d70 2024-03-18T01:17:28.887 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:28.885+0000 7fb60598a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb600102710 0x7fb60010ce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:17:28.887 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:28.886+0000 7fb60598a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb600102710 0x7fb60010ce10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51906/0 (socket says 172.21.15.29:51906) 2024-03-18T01:17:28.887 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:28.886+0000 7fb60598a700 1 -- 172.21.15.29:0/310411874 learned_addr learned my addr 172.21.15.29:0/310411874 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-18T01:17:28.887 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:28.886+0000 7fb60618b700 1 --2- 172.21.15.29:0/310411874 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb600101d70 0x7fb60010c8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:17:28.888 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:28.886+0000 7fb60598a700 1 -- 172.21.15.29:0/310411874 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb600101d70 msgr2=0x7fb60010c8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:17:28.889 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:28.886+0000 7fb60598a700 1 --2- 172.21.15.29:0/310411874 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb600101d70 0x7fb60010c8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:17:28.889 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:28.886+0000 7fb60598a700 1 -- 172.21.15.29:0/310411874 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5f0005040 con 0x7fb600102710 2024-03-18T01:17:28.889 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:28.886+0000 7fb60618b700 1 --2- 172.21.15.29:0/310411874 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb600101d70 0x7fb60010c8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T01:17:28.889 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:28.886+0000 7fb60598a700 1 --2- 172.21.15.29:0/310411874 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb600102710 0x7fb60010ce10 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fb5f0009180 tx=0x7fb5f0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:17:28.889 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:28.886+0000 7fb5f6ffd700 1 -- 172.21.15.29:0/310411874 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb5f000e040 con 0x7fb600102710 2024-03-18T01:17:28.889 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:28.886+0000 7fb6083ef700 1 -- 172.21.15.29:0/310411874 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb60010d740 con 0x7fb600102710 2024-03-18T01:17:28.889 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:28.886+0000 7fb5f6ffd700 1 -- 172.21.15.29:0/310411874 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb5f0013420 con 0x7fb600102710 2024-03-18T01:17:28.889 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:28.886+0000 7fb6083ef700 1 -- 172.21.15.29:0/310411874 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb60011a310 con 0x7fb600102710 2024-03-18T01:17:28.889 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:28.886+0000 7fb5f6ffd700 1 -- 172.21.15.29:0/310411874 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb5f001c610 con 0x7fb600102710 2024-03-18T01:17:28.889 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:28.888+0000 7fb6083ef700 1 -- 172.21.15.29:0/310411874 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5e40052b0 con 0x7fb600102710 2024-03-18T01:17:28.889 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:28.888+0000 7fb5f6ffd700 1 -- 172.21.15.29:0/310411874 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7fb5f000baf0 con 0x7fb600102710 2024-03-18T01:17:28.889 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:28.888+0000 7fb5f6ffd700 1 --2- 172.21.15.29:0/310411874 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7fb5ec0692a0 0x7fb5ec06b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:17:28.889 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:28.888+0000 7fb5f6ffd700 1 -- 172.21.15.29:0/310411874 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fb5f0087cf0 con 0x7fb600102710 2024-03-18T01:17:28.889 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:28.888+0000 7fb60618b700 1 --2- 172.21.15.29:0/310411874 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7fb5ec0692a0 0x7fb5ec06b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:17:28.890 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:28.889+0000 7fb60618b700 1 --2- 172.21.15.29:0/310411874 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7fb5ec0692a0 0x7fb5ec06b750 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fb5fc010520 tx=0x7fb5fc00b040 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:17:28.896 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:28.896+0000 7fb5f6ffd700 1 -- 172.21.15.29:0/310411874 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175520 (secure 0 0 0) 0x7fb5f0057df0 con 0x7fb600102710 2024-03-18T01:17:29.124 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:29.124+0000 7fb6083ef700 1 -- 172.21.15.29:0/310411874 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7fb5e4005560 con 0x7fb600102710 2024-03-18T01:17:29.124 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:29.124+0000 7fb5f6ffd700 1 -- 172.21.15.29:0/310411874 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fb5f0057980 con 0x7fb600102710 2024-03-18T01:17:29.125 INFO:teuthology.orchestra.run.smithi029.stdout:38654705702 2024-03-18T01:17:29.126 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:29.126+0000 7fb6083ef700 1 -- 172.21.15.29:0/310411874 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7fb5ec0692a0 msgr2=0x7fb5ec06b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:17:29.126 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:29.126+0000 7fb6083ef700 1 --2- 172.21.15.29:0/310411874 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7fb5ec0692a0 0x7fb5ec06b750 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fb5fc010520 tx=0x7fb5fc00b040 comp rx=0 tx=0).stop 2024-03-18T01:17:29.127 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:29.126+0000 7fb6083ef700 1 -- 172.21.15.29:0/310411874 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb600102710 msgr2=0x7fb60010ce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:17:29.127 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:29.126+0000 7fb6083ef700 1 --2- 172.21.15.29:0/310411874 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb600102710 0x7fb60010ce10 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fb5f0009180 tx=0x7fb5f0009e30 comp rx=0 tx=0).stop 2024-03-18T01:17:29.127 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:29.126+0000 7fb6083ef700 1 -- 172.21.15.29:0/310411874 shutdown_connections 2024-03-18T01:17:29.127 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:29.126+0000 7fb6083ef700 1 --2- 172.21.15.29:0/310411874 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7fb5ec0692a0 0x7fb5ec06b750 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:17:29.127 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:29.126+0000 7fb6083ef700 1 --2- 172.21.15.29:0/310411874 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb600101d70 0x7fb60010c8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:17:29.127 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:29.126+0000 7fb6083ef700 1 --2- 172.21.15.29:0/310411874 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb600102710 0x7fb60010ce10 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:17:29.127 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:29.126+0000 7fb6083ef700 1 -- 172.21.15.29:0/310411874 >> 172.21.15.29:0/310411874 conn(0x7fb600072bc0 msgr2=0x7fb6000fb080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:17:29.127 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:29.126+0000 7fb6083ef700 1 -- 172.21.15.29:0/310411874 shutdown_connections 2024-03-18T01:17:29.127 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:29.126+0000 7fb6083ef700 1 -- 172.21.15.29:0/310411874 wait complete. 2024-03-18T01:17:29.447 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:17:29 smithi029 ceph-mon[29331]: from='client.? 172.21.15.29:0/310411874' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-18T01:17:29.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:17:29 smithi090 ceph-mon[36989]: from='client.? 172.21.15.29:0/310411874' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-18T01:17:29.831 INFO:tasks.cephadm.ceph_manager.ceph:need seq 38654705700 got 38654705702 for osd.0 2024-03-18T01:17:29.832 DEBUG:teuthology.parallel:result is None 2024-03-18T01:17:30.256 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:30.256+0000 7f7b12226700 1 -- 172.21.15.29:0/1511894491 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7b0c1037a0 msgr2=0x7f7b0c0fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:17:30.256 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:30.256+0000 7f7b12226700 1 --2- 172.21.15.29:0/1511894491 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7b0c1037a0 0x7f7b0c0fce10 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f7b00004940 tx=0x7f7b00009e30 comp rx=0 tx=0).stop 2024-03-18T01:17:30.257 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:30.256+0000 7f7b12226700 1 -- 172.21.15.29:0/1511894491 shutdown_connections 2024-03-18T01:17:30.257 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:30.256+0000 7f7b12226700 1 --2- 172.21.15.29:0/1511894491 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7b0c1037a0 0x7f7b0c0fce10 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:17:30.257 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:30.256+0000 7f7b12226700 1 --2- 172.21.15.29:0/1511894491 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7b0c102e00 0x7f7b0c1031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:17:30.257 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:30.256+0000 7f7b12226700 1 -- 172.21.15.29:0/1511894491 >> 172.21.15.29:0/1511894491 conn(0x7f7b0c0f89b0 msgr2=0x7f7b0c0fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:17:30.257 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:30.256+0000 7f7b12226700 1 -- 172.21.15.29:0/1511894491 shutdown_connections 2024-03-18T01:17:30.257 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:30.256+0000 7f7b12226700 1 -- 172.21.15.29:0/1511894491 wait complete. 2024-03-18T01:17:30.257 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:30.256+0000 7f7b12226700 1 Processor -- start 2024-03-18T01:17:30.257 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:30.256+0000 7f7b12226700 1 -- start start 2024-03-18T01:17:30.258 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:30.256+0000 7f7b12226700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7b0c102e00 0x7f7b0c113990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:17:30.258 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:30.257+0000 7f7b12226700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7b0c1037a0 0x7f7b0c113ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:17:30.259 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:30.257+0000 7f7b12226700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b0c1145b0 con 0x7f7b0c1037a0 2024-03-18T01:17:30.259 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:30.257+0000 7f7b12226700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b0c10db40 con 0x7f7b0c102e00 2024-03-18T01:17:30.259 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:30.257+0000 7f7b0affd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7b0c1037a0 0x7f7b0c113ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:17:30.259 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:30.257+0000 7f7b0affd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7b0c1037a0 0x7f7b0c113ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51924/0 (socket says 172.21.15.29:51924) 2024-03-18T01:17:30.259 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:30.257+0000 7f7b0affd700 1 -- 172.21.15.29:0/1043118969 learned_addr learned my addr 172.21.15.29:0/1043118969 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-18T01:17:30.259 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:30.257+0000 7f7b0b7fe700 1 --2- 172.21.15.29:0/1043118969 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7b0c102e00 0x7f7b0c113990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:17:30.259 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:30.257+0000 7f7b0affd700 1 -- 172.21.15.29:0/1043118969 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7b0c102e00 msgr2=0x7f7b0c113990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:17:30.259 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:30.257+0000 7f7b0affd700 1 --2- 172.21.15.29:0/1043118969 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7b0c102e00 0x7f7b0c113990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:17:30.259 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:30.257+0000 7f7b0affd700 1 -- 172.21.15.29:0/1043118969 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b00005040 con 0x7f7b0c1037a0 2024-03-18T01:17:30.259 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:30.257+0000 7f7b0affd700 1 --2- 172.21.15.29:0/1043118969 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7b0c1037a0 0x7f7b0c113ed0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f7b00009100 tx=0x7f7b00009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:17:30.259 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:30.257+0000 7f7af3fff700 1 -- 172.21.15.29:0/1043118969 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7b0000e040 con 0x7f7b0c1037a0 2024-03-18T01:17:30.259 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:30.257+0000 7f7b12226700 1 -- 172.21.15.29:0/1043118969 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b0c10ddc0 con 0x7f7b0c1037a0 2024-03-18T01:17:30.259 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:30.257+0000 7f7af3fff700 1 -- 172.21.15.29:0/1043118969 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7b00009af0 con 0x7f7b0c1037a0 2024-03-18T01:17:30.259 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:30.257+0000 7f7af3fff700 1 -- 172.21.15.29:0/1043118969 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7b0001c3f0 con 0x7f7b0c1037a0 2024-03-18T01:17:30.260 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:30.257+0000 7f7b12226700 1 -- 172.21.15.29:0/1043118969 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b0c10e380 con 0x7f7b0c1037a0 2024-03-18T01:17:30.260 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:30.258+0000 7f7b12226700 1 -- 172.21.15.29:0/1043118969 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b0c100110 con 0x7f7b0c1037a0 2024-03-18T01:17:30.260 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:30.259+0000 7f7af3fff700 1 -- 172.21.15.29:0/1043118969 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f7b0000baf0 con 0x7f7b0c1037a0 2024-03-18T01:17:30.260 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:30.259+0000 7f7af3fff700 1 --2- 172.21.15.29:0/1043118969 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f7af40692a0 0x7f7af406b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:17:30.260 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:30.259+0000 7f7b0b7fe700 1 --2- 172.21.15.29:0/1043118969 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f7af40692a0 0x7f7af406b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:17:30.261 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:30.259+0000 7f7af3fff700 1 -- 172.21.15.29:0/1043118969 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f7b00088170 con 0x7f7b0c1037a0 2024-03-18T01:17:30.261 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:30.260+0000 7f7b0b7fe700 1 --2- 172.21.15.29:0/1043118969 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f7af40692a0 0x7f7af406b750 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f7afc009b60 tx=0x7f7afc0091a0 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:17:30.264 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:30.264+0000 7f7af3fff700 1 -- 172.21.15.29:0/1043118969 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175520 (secure 0 0 0) 0x7f7b00054740 con 0x7f7b0c1037a0 2024-03-18T01:17:30.404 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:17:30 smithi029 ceph-mon[29331]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-18T01:17:30.470 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:30.469+0000 7f7b12226700 1 -- 172.21.15.29:0/1043118969 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f7b0c10efe0 con 0x7f7b0c1037a0 2024-03-18T01:17:30.470 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:30.469+0000 7f7af3fff700 1 -- 172.21.15.29:0/1043118969 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f7b00057d60 con 0x7f7b0c1037a0 2024-03-18T01:17:30.470 INFO:teuthology.orchestra.run.smithi029.stdout:60129542176 2024-03-18T01:17:30.472 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:30.471+0000 7f7b12226700 1 -- 172.21.15.29:0/1043118969 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f7af40692a0 msgr2=0x7f7af406b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:17:30.472 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:30.471+0000 7f7b12226700 1 --2- 172.21.15.29:0/1043118969 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f7af40692a0 0x7f7af406b750 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f7afc009b60 tx=0x7f7afc0091a0 comp rx=0 tx=0).stop 2024-03-18T01:17:30.472 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:30.471+0000 7f7b12226700 1 -- 172.21.15.29:0/1043118969 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7b0c1037a0 msgr2=0x7f7b0c113ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:17:30.472 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:30.471+0000 7f7b12226700 1 --2- 172.21.15.29:0/1043118969 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7b0c1037a0 0x7f7b0c113ed0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f7b00009100 tx=0x7f7b00009e30 comp rx=0 tx=0).stop 2024-03-18T01:17:30.472 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:30.471+0000 7f7b12226700 1 -- 172.21.15.29:0/1043118969 shutdown_connections 2024-03-18T01:17:30.472 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:30.471+0000 7f7b12226700 1 --2- 172.21.15.29:0/1043118969 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f7af40692a0 0x7f7af406b750 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:17:30.472 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:30.471+0000 7f7b12226700 1 --2- 172.21.15.29:0/1043118969 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7b0c102e00 0x7f7b0c113990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:17:30.472 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:30.471+0000 7f7b12226700 1 --2- 172.21.15.29:0/1043118969 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7b0c1037a0 0x7f7b0c113ed0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:17:30.472 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:30.471+0000 7f7b12226700 1 -- 172.21.15.29:0/1043118969 >> 172.21.15.29:0/1043118969 conn(0x7f7b0c0f89b0 msgr2=0x7f7b0c0fa770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:17:30.472 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:30.472+0000 7f7b12226700 1 -- 172.21.15.29:0/1043118969 shutdown_connections 2024-03-18T01:17:30.473 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:30.472+0000 7f7b12226700 1 -- 172.21.15.29:0/1043118969 wait complete. 2024-03-18T01:17:30.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:17:30 smithi090 ceph-mon[36989]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-18T01:17:30.874 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:30.872+0000 7f405a29d700 1 -- 172.21.15.29:0/1490353138 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f40540fbc80 msgr2=0x7f40540fc050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:17:30.874 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:30.872+0000 7f405a29d700 1 --2- 172.21.15.29:0/1490353138 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f40540fbc80 0x7f40540fc050 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f403c004940 tx=0x7f403c009e30 comp rx=0 tx=0).stop 2024-03-18T01:17:30.874 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:30.872+0000 7f405a29d700 1 -- 172.21.15.29:0/1490353138 shutdown_connections 2024-03-18T01:17:30.874 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:30.872+0000 7f405a29d700 1 --2- 172.21.15.29:0/1490353138 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f40540fc590 0x7f40541092f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:17:30.874 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:30.872+0000 7f405a29d700 1 --2- 172.21.15.29:0/1490353138 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f40540fbc80 0x7f40540fc050 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:17:30.874 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:30.872+0000 7f405a29d700 1 -- 172.21.15.29:0/1490353138 >> 172.21.15.29:0/1490353138 conn(0x7f4054072bc0 msgr2=0x7f4054072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:17:30.874 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:30.872+0000 7f405a29d700 1 -- 172.21.15.29:0/1490353138 shutdown_connections 2024-03-18T01:17:30.875 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:30.872+0000 7f405a29d700 1 -- 172.21.15.29:0/1490353138 wait complete. 2024-03-18T01:17:30.875 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:30.872+0000 7f405a29d700 1 Processor -- start 2024-03-18T01:17:30.875 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:30.872+0000 7f405a29d700 1 -- start start 2024-03-18T01:17:30.875 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:30.873+0000 7f405a29d700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f40540fbc80 0x7f4054114c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:17:30.875 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:30.873+0000 7f405a29d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f40540fc590 0x7f40541151a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:17:30.875 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:30.873+0000 7f405a29d700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40541157a0 con 0x7f40540fc590 2024-03-18T01:17:30.875 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:30.873+0000 7f405a29d700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f405410fd70 con 0x7f40540fbc80 2024-03-18T01:17:30.875 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:30.873+0000 7f40537fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f40540fc590 0x7f40541151a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:17:30.875 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:30.873+0000 7f40537fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f40540fc590 0x7f40541151a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51938/0 (socket says 172.21.15.29:51938) 2024-03-18T01:17:30.875 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:30.873+0000 7f40537fe700 1 -- 172.21.15.29:0/1050628225 learned_addr learned my addr 172.21.15.29:0/1050628225 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-18T01:17:30.875 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:30.873+0000 7f4053fff700 1 --2- 172.21.15.29:0/1050628225 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f40540fbc80 0x7f4054114c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:17:30.875 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:30.873+0000 7f40537fe700 1 -- 172.21.15.29:0/1050628225 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f40540fbc80 msgr2=0x7f4054114c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:17:30.876 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:30.873+0000 7f40537fe700 1 --2- 172.21.15.29:0/1050628225 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f40540fbc80 0x7f4054114c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:17:30.876 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:30.873+0000 7f40537fe700 1 -- 172.21.15.29:0/1050628225 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f403c005040 con 0x7f40540fc590 2024-03-18T01:17:30.876 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:30.873+0000 7f40537fe700 1 --2- 172.21.15.29:0/1050628225 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f40540fc590 0x7f40541151a0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f404400c8c0 tx=0x7f40440044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:17:30.876 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:30.873+0000 7f4050ff9700 1 -- 172.21.15.29:0/1050628225 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f404400a760 con 0x7f40540fc590 2024-03-18T01:17:30.876 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:30.873+0000 7f405a29d700 1 -- 172.21.15.29:0/1050628225 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4054110050 con 0x7f40540fc590 2024-03-18T01:17:30.876 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:30.873+0000 7f4050ff9700 1 -- 172.21.15.29:0/1050628225 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f404400ad90 con 0x7f40540fc590 2024-03-18T01:17:30.876 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:30.874+0000 7f4050ff9700 1 -- 172.21.15.29:0/1050628225 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f40440113a0 con 0x7f40540fc590 2024-03-18T01:17:30.876 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:30.874+0000 7f405a29d700 1 -- 172.21.15.29:0/1050628225 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4054110610 con 0x7f40540fc590 2024-03-18T01:17:30.877 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:30.874+0000 7f405a29d700 1 -- 172.21.15.29:0/1050628225 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4054065e30 con 0x7f40540fc590 2024-03-18T01:17:30.877 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:30.875+0000 7f4050ff9700 1 -- 172.21.15.29:0/1050628225 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f4044012400 con 0x7f40540fc590 2024-03-18T01:17:30.877 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:30.875+0000 7f4050ff9700 1 --2- 172.21.15.29:0/1050628225 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f40400692a0 0x7f404006b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:17:30.877 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:30.875+0000 7f4053fff700 1 --2- 172.21.15.29:0/1050628225 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f40400692a0 0x7f404006b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:17:30.877 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:30.876+0000 7f4050ff9700 1 -- 172.21.15.29:0/1050628225 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f4044086e70 con 0x7f40540fc590 2024-03-18T01:17:30.877 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:30.876+0000 7f4053fff700 1 --2- 172.21.15.29:0/1050628225 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f40400692a0 0x7f404006b750 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f403c004f90 tx=0x7f403c009e30 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:17:30.880 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:30.880+0000 7f4050ff9700 1 -- 172.21.15.29:0/1050628225 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175520 (secure 0 0 0) 0x7f40440534e0 con 0x7f40540fc590 2024-03-18T01:17:31.076 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:31.075+0000 7f405a29d700 1 -- 172.21.15.29:0/1050628225 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f40541113a0 con 0x7f40540fc590 2024-03-18T01:17:31.076 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:31.076+0000 7f4050ff9700 1 -- 172.21.15.29:0/1050628225 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f4044056b00 con 0x7f40540fc590 2024-03-18T01:17:31.076 INFO:teuthology.orchestra.run.smithi029.stdout:146028888082 2024-03-18T01:17:31.078 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:31.077+0000 7f405a29d700 1 -- 172.21.15.29:0/1050628225 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f40400692a0 msgr2=0x7f404006b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:17:31.078 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:31.078+0000 7f405a29d700 1 --2- 172.21.15.29:0/1050628225 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f40400692a0 0x7f404006b750 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f403c004f90 tx=0x7f403c009e30 comp rx=0 tx=0).stop 2024-03-18T01:17:31.078 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:31.078+0000 7f405a29d700 1 -- 172.21.15.29:0/1050628225 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f40540fc590 msgr2=0x7f40541151a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:17:31.078 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:31.078+0000 7f405a29d700 1 --2- 172.21.15.29:0/1050628225 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f40540fc590 0x7f40541151a0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f404400c8c0 tx=0x7f40440044c0 comp rx=0 tx=0).stop 2024-03-18T01:17:31.078 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:31.078+0000 7f405a29d700 1 -- 172.21.15.29:0/1050628225 shutdown_connections 2024-03-18T01:17:31.078 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:31.078+0000 7f405a29d700 1 --2- 172.21.15.29:0/1050628225 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f40400692a0 0x7f404006b750 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:17:31.079 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:31.078+0000 7f405a29d700 1 --2- 172.21.15.29:0/1050628225 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f40540fbc80 0x7f4054114c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:17:31.079 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:31.078+0000 7f405a29d700 1 --2- 172.21.15.29:0/1050628225 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f40540fc590 0x7f40541151a0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:17:31.079 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:31.078+0000 7f405a29d700 1 -- 172.21.15.29:0/1050628225 >> 172.21.15.29:0/1050628225 conn(0x7f4054072bc0 msgr2=0x7f4054105ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:17:31.079 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:31.078+0000 7f405a29d700 1 -- 172.21.15.29:0/1050628225 shutdown_connections 2024-03-18T01:17:31.079 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:31.078+0000 7f405a29d700 1 -- 172.21.15.29:0/1050628225 wait complete. 2024-03-18T01:17:31.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:17:31 smithi090 ceph-mon[36989]: from='client.? 172.21.15.29:0/1043118969' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-18T01:17:31.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:17:31 smithi090 ceph-mon[36989]: from='client.? 172.21.15.29:0/1050628225' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-18T01:17:31.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:17:31 smithi029 ceph-mon[29331]: from='client.? 172.21.15.29:0/1043118969' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-18T01:17:31.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:17:31 smithi029 ceph-mon[29331]: from='client.? 172.21.15.29:0/1050628225' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-18T01:17:31.943 INFO:tasks.cephadm.ceph_manager.ceph:need seq 60129542174 got 60129542176 for osd.1 2024-03-18T01:17:31.943 DEBUG:teuthology.parallel:result is None 2024-03-18T01:17:32.465 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:17:32 smithi029 ceph-mon[29331]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-18T01:17:32.498 INFO:tasks.cephadm.ceph_manager.ceph:need seq 146028888080 got 146028888082 for osd.4 2024-03-18T01:17:32.498 DEBUG:teuthology.parallel:result is None 2024-03-18T01:17:32.525 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:17:32 smithi090 ceph-mon[36989]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-18T01:17:33.531 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:33.528+0000 7fc44383f700 1 -- 172.21.15.29:0/1773136761 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc43c102e10 msgr2=0x7fc43c1031e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:17:33.531 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:33.528+0000 7fc44383f700 1 --2- 172.21.15.29:0/1773136761 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc43c102e10 0x7fc43c1031e0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fc42c004940 tx=0x7fc42c009e30 comp rx=0 tx=0).stop 2024-03-18T01:17:33.531 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:33.528+0000 7fc44383f700 1 -- 172.21.15.29:0/1773136761 shutdown_connections 2024-03-18T01:17:33.531 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:33.528+0000 7fc44383f700 1 --2- 172.21.15.29:0/1773136761 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc43c1037b0 0x7fc43c0fce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:17:33.531 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:33.529+0000 7fc44383f700 1 --2- 172.21.15.29:0/1773136761 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc43c102e10 0x7fc43c1031e0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:17:33.531 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:33.529+0000 7fc44383f700 1 -- 172.21.15.29:0/1773136761 >> 172.21.15.29:0/1773136761 conn(0x7fc43c0f8980 msgr2=0x7fc43c0fad90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:17:33.531 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:33.529+0000 7fc44383f700 1 -- 172.21.15.29:0/1773136761 shutdown_connections 2024-03-18T01:17:33.531 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:33.529+0000 7fc44383f700 1 -- 172.21.15.29:0/1773136761 wait complete. 2024-03-18T01:17:33.532 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:33.529+0000 7fc44383f700 1 Processor -- start 2024-03-18T01:17:33.532 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:33.529+0000 7fc44383f700 1 -- start start 2024-03-18T01:17:33.532 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:33.529+0000 7fc44383f700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc43c102e10 0x7fc43c113ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:17:33.532 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:33.529+0000 7fc44383f700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc43c1037b0 0x7fc43c1140e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:17:33.532 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:33.529+0000 7fc44383f700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc43c114730 con 0x7fc43c1037b0 2024-03-18T01:17:33.532 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:33.529+0000 7fc44383f700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc43c10db40 con 0x7fc43c102e10 2024-03-18T01:17:33.532 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:33.529+0000 7fc440dda700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc43c1037b0 0x7fc43c1140e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:17:33.532 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:33.530+0000 7fc440dda700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc43c1037b0 0x7fc43c1140e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39360/0 (socket says 172.21.15.29:39360) 2024-03-18T01:17:33.532 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:33.530+0000 7fc440dda700 1 -- 172.21.15.29:0/2796078207 learned_addr learned my addr 172.21.15.29:0/2796078207 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-18T01:17:33.532 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:33.530+0000 7fc4415db700 1 --2- 172.21.15.29:0/2796078207 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc43c102e10 0x7fc43c113ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:17:33.532 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:33.530+0000 7fc440dda700 1 -- 172.21.15.29:0/2796078207 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc43c102e10 msgr2=0x7fc43c113ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:17:33.532 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:33.530+0000 7fc440dda700 1 --2- 172.21.15.29:0/2796078207 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc43c102e10 0x7fc43c113ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:17:33.532 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:33.530+0000 7fc440dda700 1 -- 172.21.15.29:0/2796078207 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc42c005040 con 0x7fc43c1037b0 2024-03-18T01:17:33.532 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:33.530+0000 7fc4415db700 1 --2- 172.21.15.29:0/2796078207 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc43c102e10 0x7fc43c113ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T01:17:33.533 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:33.530+0000 7fc440dda700 1 --2- 172.21.15.29:0/2796078207 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc43c1037b0 0x7fc43c1140e0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fc438002a50 tx=0x7fc438004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:17:33.533 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:33.530+0000 7fc431ffb700 1 -- 172.21.15.29:0/2796078207 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc438007cc0 con 0x7fc43c1037b0 2024-03-18T01:17:33.533 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:33.530+0000 7fc431ffb700 1 -- 172.21.15.29:0/2796078207 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc43800f040 con 0x7fc43c1037b0 2024-03-18T01:17:33.533 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:33.530+0000 7fc44383f700 1 -- 172.21.15.29:0/2796078207 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc43c10de20 con 0x7fc43c1037b0 2024-03-18T01:17:33.533 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:33.530+0000 7fc431ffb700 1 -- 172.21.15.29:0/2796078207 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc43800daf0 con 0x7fc43c1037b0 2024-03-18T01:17:33.533 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:33.530+0000 7fc44383f700 1 -- 172.21.15.29:0/2796078207 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc43c10e3e0 con 0x7fc43c1037b0 2024-03-18T01:17:33.534 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:33.531+0000 7fc44383f700 1 -- 172.21.15.29:0/2796078207 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc43c100120 con 0x7fc43c1037b0 2024-03-18T01:17:33.534 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:33.532+0000 7fc431ffb700 1 -- 172.21.15.29:0/2796078207 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7fc438009810 con 0x7fc43c1037b0 2024-03-18T01:17:33.534 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:33.532+0000 7fc431ffb700 1 --2- 172.21.15.29:0/2796078207 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7fc4280692a0 0x7fc42806b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:17:33.534 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:33.532+0000 7fc431ffb700 1 -- 172.21.15.29:0/2796078207 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fc438086bf0 con 0x7fc43c1037b0 2024-03-18T01:17:33.534 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:33.532+0000 7fc4415db700 1 --2- 172.21.15.29:0/2796078207 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7fc4280692a0 0x7fc42806b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:17:33.534 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:33.533+0000 7fc4415db700 1 --2- 172.21.15.29:0/2796078207 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7fc4280692a0 0x7fc42806b750 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fc42c004560 tx=0x7fc42c009e30 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:17:33.537 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:33.537+0000 7fc431ffb700 1 -- 172.21.15.29:0/2796078207 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175520 (secure 0 0 0) 0x7fc4380531e0 con 0x7fc43c1037b0 2024-03-18T01:17:33.729 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:33.728+0000 7fc44383f700 1 -- 172.21.15.29:0/2796078207 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7fc43c10f220 con 0x7fc43c1037b0 2024-03-18T01:17:33.729 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:33.728+0000 7fc431ffb700 1 -- 172.21.15.29:0/2796078207 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7fc438056800 con 0x7fc43c1037b0 2024-03-18T01:17:33.729 INFO:teuthology.orchestra.run.smithi029.stdout:167503724558 2024-03-18T01:17:33.731 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:33.730+0000 7fc44383f700 1 -- 172.21.15.29:0/2796078207 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7fc4280692a0 msgr2=0x7fc42806b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:17:33.731 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:33.730+0000 7fc44383f700 1 --2- 172.21.15.29:0/2796078207 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7fc4280692a0 0x7fc42806b750 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fc42c004560 tx=0x7fc42c009e30 comp rx=0 tx=0).stop 2024-03-18T01:17:33.731 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:33.730+0000 7fc44383f700 1 -- 172.21.15.29:0/2796078207 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc43c1037b0 msgr2=0x7fc43c1140e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:17:33.731 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:33.730+0000 7fc44383f700 1 --2- 172.21.15.29:0/2796078207 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc43c1037b0 0x7fc43c1140e0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fc438002a50 tx=0x7fc438004600 comp rx=0 tx=0).stop 2024-03-18T01:17:33.731 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:33.730+0000 7fc44383f700 1 -- 172.21.15.29:0/2796078207 shutdown_connections 2024-03-18T01:17:33.731 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:33.730+0000 7fc44383f700 1 --2- 172.21.15.29:0/2796078207 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7fc4280692a0 0x7fc42806b750 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:17:33.731 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:33.730+0000 7fc44383f700 1 --2- 172.21.15.29:0/2796078207 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc43c102e10 0x7fc43c113ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:17:33.731 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:33.730+0000 7fc44383f700 1 --2- 172.21.15.29:0/2796078207 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc43c1037b0 0x7fc43c1140e0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:17:33.731 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:33.730+0000 7fc44383f700 1 -- 172.21.15.29:0/2796078207 >> 172.21.15.29:0/2796078207 conn(0x7fc43c0f8980 msgr2=0x7fc43c107400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:17:33.732 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:33.731+0000 7fc44383f700 1 -- 172.21.15.29:0/2796078207 shutdown_connections 2024-03-18T01:17:33.732 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:33.731+0000 7fc44383f700 1 -- 172.21.15.29:0/2796078207 wait complete. 2024-03-18T01:17:34.296 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:17:34 smithi029 ceph-mon[29331]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-18T01:17:34.297 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:17:34 smithi029 ceph-mon[29331]: from='client.? 172.21.15.29:0/2796078207' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-18T01:17:34.298 INFO:tasks.cephadm.ceph_manager.ceph:need seq 167503724555 got 167503724558 for osd.5 2024-03-18T01:17:34.298 DEBUG:teuthology.parallel:result is None 2024-03-18T01:17:34.298 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-03-18T01:17:34.298 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 37f51182-e4c4-11ee-95c9-87774f69a715 -- ceph pg dump --format=json 2024-03-18T01:17:34.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:17:34 smithi090 ceph-mon[36989]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-18T01:17:34.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:17:34 smithi090 ceph-mon[36989]: from='client.? 172.21.15.29:0/2796078207' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-18T01:17:34.561 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/37f51182-e4c4-11ee-95c9-87774f69a715/mon.smithi029/config 2024-03-18T01:17:36.141 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:36.141+0000 7f430d25e700 1 -- 172.21.15.29:0/3537637337 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4308102f70 msgr2=0x7f4308103340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:17:36.141 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:36.141+0000 7f430d25e700 1 --2- 172.21.15.29:0/3537637337 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4308102f70 0x7f4308103340 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f4300009ce0 tx=0x7f4300004d10 comp rx=0 tx=0).stop 2024-03-18T01:17:36.142 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:36.141+0000 7f430d25e700 1 -- 172.21.15.29:0/3537637337 shutdown_connections 2024-03-18T01:17:36.142 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:36.141+0000 7f430d25e700 1 --2- 172.21.15.29:0/3537637337 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f43080fb3b0 0x7f43080fb820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:17:36.142 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:36.141+0000 7f430d25e700 1 --2- 172.21.15.29:0/3537637337 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4308102f70 0x7f4308103340 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:17:36.142 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:36.141+0000 7f430d25e700 1 -- 172.21.15.29:0/3537637337 >> 172.21.15.29:0/3537637337 conn(0x7f4308072310 msgr2=0x7f4308072710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:17:36.142 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:36.141+0000 7f430d25e700 1 -- 172.21.15.29:0/3537637337 shutdown_connections 2024-03-18T01:17:36.142 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:36.141+0000 7f430d25e700 1 -- 172.21.15.29:0/3537637337 wait complete. 2024-03-18T01:17:36.142 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:36.141+0000 7f430d25e700 1 Processor -- start 2024-03-18T01:17:36.142 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:36.141+0000 7f430d25e700 1 -- start start 2024-03-18T01:17:36.144 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:36.141+0000 7f430d25e700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f43080fb3b0 0x7f43080fc590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:17:36.144 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:36.141+0000 7f430d25e700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4308102f70 0x7f43080fcad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:17:36.144 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:36.142+0000 7f430d25e700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43080fd010 con 0x7f4308102f70 2024-03-18T01:17:36.144 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:36.142+0000 7f430d25e700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43080fd180 con 0x7f43080fb3b0 2024-03-18T01:17:36.144 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:36.142+0000 7f430659c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4308102f70 0x7f43080fcad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:17:36.144 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:36.142+0000 7f430659c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4308102f70 0x7f43080fcad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39372/0 (socket says 172.21.15.29:39372) 2024-03-18T01:17:36.144 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:36.142+0000 7f430659c700 1 -- 172.21.15.29:0/1216009125 learned_addr learned my addr 172.21.15.29:0/1216009125 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-18T01:17:36.144 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:36.142+0000 7f4306d9d700 1 --2- 172.21.15.29:0/1216009125 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f43080fb3b0 0x7f43080fc590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:17:36.144 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:36.142+0000 7f430659c700 1 -- 172.21.15.29:0/1216009125 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f43080fb3b0 msgr2=0x7f43080fc590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:17:36.144 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:36.142+0000 7f430659c700 1 --2- 172.21.15.29:0/1216009125 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f43080fb3b0 0x7f43080fc590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:17:36.144 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:36.142+0000 7f430659c700 1 -- 172.21.15.29:0/1216009125 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4300009940 con 0x7f4308102f70 2024-03-18T01:17:36.144 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:36.142+0000 7f430659c700 1 --2- 172.21.15.29:0/1216009125 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4308102f70 0x7f43080fcad0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f42f000b790 tx=0x7f42f0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:17:36.144 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:36.142+0000 7f42ff7fe700 1 -- 172.21.15.29:0/1216009125 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f42f000f6f0 con 0x7f4308102f70 2024-03-18T01:17:36.145 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:36.142+0000 7f430d25e700 1 -- 172.21.15.29:0/1216009125 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f430811a030 con 0x7f4308102f70 2024-03-18T01:17:36.145 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:36.142+0000 7f42ff7fe700 1 -- 172.21.15.29:0/1216009125 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f42f000fd20 con 0x7f4308102f70 2024-03-18T01:17:36.145 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:36.142+0000 7f42ff7fe700 1 -- 172.21.15.29:0/1216009125 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f42f00103a0 con 0x7f4308102f70 2024-03-18T01:17:36.145 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:36.142+0000 7f430d25e700 1 -- 172.21.15.29:0/1216009125 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f430811a380 con 0x7f4308102f70 2024-03-18T01:17:36.145 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:36.143+0000 7f430d25e700 1 -- 172.21.15.29:0/1216009125 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43080fe6d0 con 0x7f4308102f70 2024-03-18T01:17:36.145 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:36.144+0000 7f42ff7fe700 1 -- 172.21.15.29:0/1216009125 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f42f0011400 con 0x7f4308102f70 2024-03-18T01:17:36.146 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:36.144+0000 7f42ff7fe700 1 --2- 172.21.15.29:0/1216009125 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f42f40692a0 0x7f42f406b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:17:36.146 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:36.144+0000 7f42ff7fe700 1 -- 172.21.15.29:0/1216009125 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f42f0086c20 con 0x7f4308102f70 2024-03-18T01:17:36.146 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:36.144+0000 7f4306d9d700 1 --2- 172.21.15.29:0/1216009125 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f42f40692a0 0x7f42f406b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:17:36.146 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:36.145+0000 7f4306d9d700 1 --2- 172.21.15.29:0/1216009125 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f42f40692a0 0x7f42f406b750 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f430000b740 tx=0x7f4300004d10 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:17:36.149 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:36.148+0000 7f42ff7fe700 1 -- 172.21.15.29:0/1216009125 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175520 (secure 0 0 0) 0x7f42f0053290 con 0x7f4308102f70 2024-03-18T01:17:36.336 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:36.336+0000 7f430d25e700 1 -- 172.21.15.29:0/1216009125 --> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f430810a080 con 0x7f42f40692a0 2024-03-18T01:17:36.337 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:36.337+0000 7f42ff7fe700 1 -- 172.21.15.29:0/1216009125 <== mgr.14208 v2:172.21.15.29:6800/1641454714 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+28262 (secure 0 0 0) 0x7f430810a080 con 0x7f42f40692a0 2024-03-18T01:17:36.338 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-18T01:17:36.340 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:36.339+0000 7f430d25e700 1 -- 172.21.15.29:0/1216009125 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f42f40692a0 msgr2=0x7f42f406b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:17:36.340 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:36.339+0000 7f430d25e700 1 --2- 172.21.15.29:0/1216009125 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f42f40692a0 0x7f42f406b750 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f430000b740 tx=0x7f4300004d10 comp rx=0 tx=0).stop 2024-03-18T01:17:36.340 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:36.339+0000 7f430d25e700 1 -- 172.21.15.29:0/1216009125 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4308102f70 msgr2=0x7f43080fcad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:17:36.340 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:36.339+0000 7f430d25e700 1 --2- 172.21.15.29:0/1216009125 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4308102f70 0x7f43080fcad0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f42f000b790 tx=0x7f42f0009250 comp rx=0 tx=0).stop 2024-03-18T01:17:36.340 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:36.339+0000 7f430d25e700 1 -- 172.21.15.29:0/1216009125 shutdown_connections 2024-03-18T01:17:36.340 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:36.339+0000 7f430d25e700 1 --2- 172.21.15.29:0/1216009125 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f42f40692a0 0x7f42f406b750 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:17:36.340 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:36.339+0000 7f430d25e700 1 --2- 172.21.15.29:0/1216009125 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f43080fb3b0 0x7f43080fc590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:17:36.340 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:36.339+0000 7f430d25e700 1 --2- 172.21.15.29:0/1216009125 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4308102f70 0x7f43080fcad0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:17:36.340 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:36.339+0000 7f430d25e700 1 -- 172.21.15.29:0/1216009125 >> 172.21.15.29:0/1216009125 conn(0x7f4308072310 msgr2=0x7f43080f9580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:17:36.340 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:36.339+0000 7f430d25e700 1 -- 172.21.15.29:0/1216009125 shutdown_connections 2024-03-18T01:17:36.340 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:36.339+0000 7f430d25e700 1 -- 172.21.15.29:0/1216009125 wait complete. 2024-03-18T01:17:36.341 INFO:teuthology.orchestra.run.smithi029.stderr:dumped all 2024-03-18T01:17:36.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:17:36 smithi090 ceph-mon[36989]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-18T01:17:36.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:17:36 smithi029 ceph-mon[29331]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-18T01:17:36.985 INFO:teuthology.orchestra.run.smithi029.stdout:{"pg_ready":true,"pg_map":{"version":160,"stamp":"2024-03-18T01:17:34.976192+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":3,"kb":562470912,"kb_used":1787600,"kb_used_data":3872,"kb_used_omap":0,"kb_used_meta":1783680,"kb_avail":560683312,"statfs":{"total":575970213888,"available":574139711488,"internally_reserved":0,"allocated":3964928,"data_stored":2605601,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1826488320},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002006"},"pg_stats":[{"pgid":"1.0","version":"23'117","reported_seq":190,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T01:16:41.544296+0000","last_change":"2024-03-18T01:16:17.054452+0000","last_active":"2024-03-18T01:16:41.544296+0000","last_peered":"2024-03-18T01:16:41.544296+0000","last_clean":"2024-03-18T01:16:41.544296+0000","last_became_active":"2024-03-18T01:16:17.054149+0000","last_became_peered":"2024-03-18T01:16:17.054149+0000","last_unstale":"2024-03-18T01:16:41.544296+0000","last_undegraded":"2024-03-18T01:16:41.544296+0000","last_fullsized":"2024-03-18T01:16:41.544296+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:15:31.263848+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:15:31.263848+0000","last_clean_scrub_stamp":"2024-03-18T01:15:31.263848+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:33:09.523250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":5,"up_from":39,"seq":167503724558,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297644,"kb_used_data":356,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93447508,"statfs":{"total":95995035648,"available":95690248192,"internally_reserved":0,"allocated":364544,"data_stored":139193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58099999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68999999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67700000000000005}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.623}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51700000000000002}]}]},{"osd":4,"up_from":34,"seq":146028888083,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297644,"kb_used_data":356,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93447508,"statfs":{"total":95995035648,"available":95690248192,"internally_reserved":0,"allocated":364544,"data_stored":139193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 01:17:19 2024","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.83299999999999996,"5min":0.83299999999999996,"15min":0.83299999999999996},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.80900000000000005},"last":0.61899999999999999}]},{"osd":1,"last update":"Mon Mar 18 01:17:19 2024","interfaces":[{"interface":"back","average":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.64600000000000002},{"interface":"front","average":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.69699999999999995,"5min":0.69699999999999995,"15min":0.69699999999999995},"last":0.39600000000000002}]},{"osd":2,"last update":"Mon Mar 18 01:17:19 2024","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"last":0.503},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"last":0.42399999999999999}]},{"osd":3,"last update":"Mon Mar 18 01:17:19 2024","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.626,"5min":0.626,"15min":0.626},"last":0.376},{"interface":"front","average":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"last":0.436}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46200000000000002}]}]},{"osd":3,"up_from":26,"seq":111669149719,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298224,"kb_used_data":936,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93446928,"statfs":{"total":95995035648,"available":95689654272,"internally_reserved":0,"allocated":958464,"data_stored":729561,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 01:16:55 2024","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.81499999999999995,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.88900000000000001,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.48799999999999999}]},{"osd":1,"last update":"Mon Mar 18 01:16:55 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.86599999999999999},"last":0.501},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.874,"5min":0.874,"15min":0.874},"last":0.53700000000000003}]},{"osd":2,"last update":"Mon Mar 18 01:16:55 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"last":0.46899999999999997}]},{"osd":4,"last update":"Mon Mar 18 01:17:17 2024","interfaces":[{"interface":"back","average":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"last":0.30599999999999999},{"interface":"front","average":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"last":0.38100000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35499999999999998}]}]},{"osd":2,"up_from":19,"seq":81604378652,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297640,"kb_used_data":352,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93447512,"statfs":{"total":95995035648,"available":95690252288,"internally_reserved":0,"allocated":360448,"data_stored":138532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 01:16:33 2024","interfaces":[{"interface":"back","average":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"last":0.40000000000000002},{"interface":"front","average":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"last":0.27600000000000002}]},{"osd":1,"last update":"Mon Mar 18 01:16:33 2024","interfaces":[{"interface":"back","average":{"1min":0.377,"5min":0.377,"15min":0.377},"min":{"1min":0.246,"5min":0.246,"15min":0.246},"max":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"last":0.38100000000000001},{"interface":"front","average":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"last":0.35599999999999998}]},{"osd":3,"last update":"Mon Mar 18 01:16:56 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.67200000000000004},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"last":0.56000000000000005}]},{"osd":4,"last update":"Mon Mar 18 01:17:17 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":0.81899999999999995,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.70799999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55300000000000005}]}]},{"osd":0,"up_from":9,"seq":38654705704,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298224,"kb_used_data":936,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93446928,"statfs":{"total":95995035648,"available":95689654272,"internally_reserved":0,"allocated":958464,"data_stored":729561,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Mar 18 01:17:06 2024","interfaces":[{"interface":"back","average":{"1min":0.38800000000000001,"5min":0.32800000000000001,"15min":0.318},"min":{"1min":0.28699999999999998,"5min":0.22,"15min":0.22},"max":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"last":0.498},{"interface":"front","average":{"1min":0.378,"5min":0.33500000000000002,"15min":0.32800000000000001},"min":{"1min":0.27800000000000002,"5min":0.214,"15min":0.214},"max":{"1min":0.49299999999999999,"5min":0.54500000000000004,"15min":0.54500000000000004},"last":0.58699999999999997}]},{"osd":2,"last update":"Mon Mar 18 01:17:31 2024","interfaces":[{"interface":"back","average":{"1min":0.38600000000000001,"5min":0.36099999999999999,"15min":0.35699999999999998},"min":{"1min":0.27300000000000002,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.39500000000000002,"5min":0.40100000000000002,"15min":0.40200000000000002},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.55900000000000005,"5min":0.57999999999999996,"15min":0.57999999999999996},"last":0.54600000000000004}]},{"osd":3,"last update":"Mon Mar 18 01:16:57 2024","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.753,"5min":0.753,"15min":0.753},"last":0.77400000000000002},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"last":0.66000000000000003}]},{"osd":4,"last update":"Mon Mar 18 01:17:18 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"last":0.69999999999999996},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.71099999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73999999999999999}]}]},{"osd":1,"up_from":14,"seq":60129542177,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298224,"kb_used_data":936,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93446928,"statfs":{"total":95995035648,"available":95689654272,"internally_reserved":0,"allocated":958464,"data_stored":729561,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 01:17:02 2024","interfaces":[{"interface":"back","average":{"1min":0.36199999999999999,"5min":0.31900000000000001,"15min":0.312},"min":{"1min":0.26600000000000001,"5min":0.17599999999999999,"15min":0.17599999999999999},"max":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"last":0.30399999999999999},{"interface":"front","average":{"1min":0.36699999999999999,"5min":0.33300000000000002,"15min":0.32700000000000001},"min":{"1min":0.24099999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":0.53700000000000003,"5min":0.54300000000000004,"15min":0.54300000000000004},"last":0.25900000000000001}]},{"osd":2,"last update":"Mon Mar 18 01:16:34 2024","interfaces":[{"interface":"back","average":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"last":0.28199999999999997},{"interface":"front","average":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"last":0.317}]},{"osd":3,"last update":"Mon Mar 18 01:16:55 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.377},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"last":0.46400000000000002}]},{"osd":4,"last update":"Mon Mar 18 01:17:17 2024","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.45400000000000001},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.47299999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40200000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-18T01:17:36.986 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 37f51182-e4c4-11ee-95c9-87774f69a715 -- ceph pg dump --format=json 2024-03-18T01:17:37.257 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/37f51182-e4c4-11ee-95c9-87774f69a715/mon.smithi029/config 2024-03-18T01:17:38.427 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:17:38 smithi090 ceph-mon[36989]: from='client.14426 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-18T01:17:38.427 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:17:38 smithi090 ceph-mon[36989]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-18T01:17:38.461 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:17:38 smithi029 ceph-mon[29331]: from='client.14426 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-18T01:17:38.461 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:17:38 smithi029 ceph-mon[29331]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-18T01:17:38.969 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:38.968+0000 7f9a343bc700 1 -- 172.21.15.29:0/1126366230 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9a2c0fb540 msgr2=0x7f9a2c0fb9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:17:38.969 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:38.968+0000 7f9a343bc700 1 --2- 172.21.15.29:0/1126366230 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9a2c0fb540 0x7f9a2c0fb9b0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f9a1c004940 tx=0x7f9a1c009e30 comp rx=0 tx=0).stop 2024-03-18T01:17:38.969 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:38.968+0000 7f9a343bc700 1 -- 172.21.15.29:0/1126366230 shutdown_connections 2024-03-18T01:17:38.969 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:38.968+0000 7f9a343bc700 1 --2- 172.21.15.29:0/1126366230 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9a2c0fb540 0x7f9a2c0fb9b0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:17:38.969 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:38.968+0000 7f9a343bc700 1 --2- 172.21.15.29:0/1126366230 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9a2c0fdd80 0x7f9a2c0fe150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:17:38.969 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:38.968+0000 7f9a343bc700 1 -- 172.21.15.29:0/1126366230 >> 172.21.15.29:0/1126366230 conn(0x7f9a2c071530 msgr2=0x7f9a2c071930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:17:38.969 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:38.968+0000 7f9a343bc700 1 -- 172.21.15.29:0/1126366230 shutdown_connections 2024-03-18T01:17:38.969 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:38.968+0000 7f9a343bc700 1 -- 172.21.15.29:0/1126366230 wait complete. 2024-03-18T01:17:38.971 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:38.968+0000 7f9a343bc700 1 Processor -- start 2024-03-18T01:17:38.971 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:38.969+0000 7f9a343bc700 1 -- start start 2024-03-18T01:17:38.971 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:38.969+0000 7f9a343bc700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9a2c0fb540 0x7f9a2c115c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:17:38.971 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:38.969+0000 7f9a343bc700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9a2c0fdd80 0x7f9a2c116190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:17:38.971 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:38.969+0000 7f9a343bc700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a2c116870 con 0x7f9a2c0fdd80 2024-03-18T01:17:38.972 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:38.969+0000 7f9a343bc700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a2c10fdc0 con 0x7f9a2c0fb540 2024-03-18T01:17:38.972 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:38.969+0000 7f9a31957700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9a2c0fdd80 0x7f9a2c116190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:17:38.972 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:38.969+0000 7f9a31957700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9a2c0fdd80 0x7f9a2c116190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39390/0 (socket says 172.21.15.29:39390) 2024-03-18T01:17:38.972 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:38.969+0000 7f9a31957700 1 -- 172.21.15.29:0/2804368218 learned_addr learned my addr 172.21.15.29:0/2804368218 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-18T01:17:38.972 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:38.970+0000 7f9a32158700 1 --2- 172.21.15.29:0/2804368218 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9a2c0fb540 0x7f9a2c115c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:17:38.972 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:38.970+0000 7f9a31957700 1 -- 172.21.15.29:0/2804368218 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9a2c0fb540 msgr2=0x7f9a2c115c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:17:38.972 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:38.970+0000 7f9a31957700 1 --2- 172.21.15.29:0/2804368218 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9a2c0fb540 0x7f9a2c115c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:17:38.972 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:38.970+0000 7f9a31957700 1 -- 172.21.15.29:0/2804368218 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a1c005040 con 0x7f9a2c0fdd80 2024-03-18T01:17:38.972 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:38.970+0000 7f9a32158700 1 --2- 172.21.15.29:0/2804368218 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9a2c0fb540 0x7f9a2c115c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T01:17:38.972 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:38.970+0000 7f9a31957700 1 --2- 172.21.15.29:0/2804368218 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9a2c0fdd80 0x7f9a2c116190 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f9a1c009150 tx=0x7f9a1c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:17:38.972 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:38.970+0000 7f9a22ffd700 1 -- 172.21.15.29:0/2804368218 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9a1c00e040 con 0x7f9a2c0fdd80 2024-03-18T01:17:38.972 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:38.970+0000 7f9a343bc700 1 -- 172.21.15.29:0/2804368218 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a2c10ffe0 con 0x7f9a2c0fdd80 2024-03-18T01:17:38.972 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:38.970+0000 7f9a22ffd700 1 -- 172.21.15.29:0/2804368218 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9a1c013420 con 0x7f9a2c0fdd80 2024-03-18T01:17:38.973 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:38.970+0000 7f9a22ffd700 1 -- 172.21.15.29:0/2804368218 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9a1c01c630 con 0x7f9a2c0fdd80 2024-03-18T01:17:38.973 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:38.970+0000 7f9a343bc700 1 -- 172.21.15.29:0/2804368218 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a2c110520 con 0x7f9a2c0fdd80 2024-03-18T01:17:38.973 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:38.971+0000 7f9a343bc700 1 -- 172.21.15.29:0/2804368218 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a2c1073f0 con 0x7f9a2c0fdd80 2024-03-18T01:17:38.973 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:38.972+0000 7f9a22ffd700 1 -- 172.21.15.29:0/2804368218 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f9a1c013590 con 0x7f9a2c0fdd80 2024-03-18T01:17:38.973 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:38.972+0000 7f9a22ffd700 1 --2- 172.21.15.29:0/2804368218 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f9a180692a0 0x7f9a1806b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:17:38.973 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:38.972+0000 7f9a22ffd700 1 -- 172.21.15.29:0/2804368218 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f9a1c0880d0 con 0x7f9a2c0fdd80 2024-03-18T01:17:38.974 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:38.972+0000 7f9a32158700 1 --2- 172.21.15.29:0/2804368218 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f9a180692a0 0x7f9a1806b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:17:38.974 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:38.972+0000 7f9a32158700 1 --2- 172.21.15.29:0/2804368218 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f9a180692a0 0x7f9a1806b750 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f9a28006fd0 tx=0x7f9a28008040 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:17:38.976 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:38.976+0000 7f9a22ffd700 1 -- 172.21.15.29:0/2804368218 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175520 (secure 0 0 0) 0x7f9a1c058280 con 0x7f9a2c0fdd80 2024-03-18T01:17:39.164 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:39.163+0000 7f9a343bc700 1 -- 172.21.15.29:0/2804368218 --> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f9a2c10c2d0 con 0x7f9a180692a0 2024-03-18T01:17:39.165 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:39.165+0000 7f9a22ffd700 1 -- 172.21.15.29:0/2804368218 <== mgr.14208 v2:172.21.15.29:6800/1641454714 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+28274 (secure 0 0 0) 0x7f9a2c10c2d0 con 0x7f9a180692a0 2024-03-18T01:17:39.166 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-18T01:17:39.167 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:39.167+0000 7f9a343bc700 1 -- 172.21.15.29:0/2804368218 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f9a180692a0 msgr2=0x7f9a1806b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:17:39.167 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:39.167+0000 7f9a343bc700 1 --2- 172.21.15.29:0/2804368218 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f9a180692a0 0x7f9a1806b750 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f9a28006fd0 tx=0x7f9a28008040 comp rx=0 tx=0).stop 2024-03-18T01:17:39.167 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:39.167+0000 7f9a343bc700 1 -- 172.21.15.29:0/2804368218 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9a2c0fdd80 msgr2=0x7f9a2c116190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:17:39.168 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:39.167+0000 7f9a343bc700 1 --2- 172.21.15.29:0/2804368218 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9a2c0fdd80 0x7f9a2c116190 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f9a1c009150 tx=0x7f9a1c009e30 comp rx=0 tx=0).stop 2024-03-18T01:17:39.168 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:39.167+0000 7f9a343bc700 1 -- 172.21.15.29:0/2804368218 shutdown_connections 2024-03-18T01:17:39.168 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:39.167+0000 7f9a343bc700 1 --2- 172.21.15.29:0/2804368218 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f9a180692a0 0x7f9a1806b750 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:17:39.168 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:39.167+0000 7f9a343bc700 1 --2- 172.21.15.29:0/2804368218 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9a2c0fb540 0x7f9a2c115c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:17:39.168 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:39.167+0000 7f9a343bc700 1 --2- 172.21.15.29:0/2804368218 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9a2c0fdd80 0x7f9a2c116190 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:17:39.168 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:39.167+0000 7f9a343bc700 1 -- 172.21.15.29:0/2804368218 >> 172.21.15.29:0/2804368218 conn(0x7f9a2c071530 msgr2=0x7f9a2c0f98d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:17:39.168 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:39.167+0000 7f9a343bc700 1 -- 172.21.15.29:0/2804368218 shutdown_connections 2024-03-18T01:17:39.168 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:39.167+0000 7f9a343bc700 1 -- 172.21.15.29:0/2804368218 wait complete. 2024-03-18T01:17:39.168 INFO:teuthology.orchestra.run.smithi029.stderr:dumped all 2024-03-18T01:17:39.805 INFO:teuthology.orchestra.run.smithi029.stdout:{"pg_ready":true,"pg_map":{"version":162,"stamp":"2024-03-18T01:17:38.976784+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":3,"kb":562470912,"kb_used":1787600,"kb_used_data":3872,"kb_used_omap":0,"kb_used_meta":1783680,"kb_avail":560683312,"statfs":{"total":575970213888,"available":574139711488,"internally_reserved":0,"allocated":3964928,"data_stored":2605601,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1826488320},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001973"},"pg_stats":[{"pgid":"1.0","version":"23'117","reported_seq":190,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T01:16:41.544296+0000","last_change":"2024-03-18T01:16:17.054452+0000","last_active":"2024-03-18T01:16:41.544296+0000","last_peered":"2024-03-18T01:16:41.544296+0000","last_clean":"2024-03-18T01:16:41.544296+0000","last_became_active":"2024-03-18T01:16:17.054149+0000","last_became_peered":"2024-03-18T01:16:17.054149+0000","last_unstale":"2024-03-18T01:16:41.544296+0000","last_undegraded":"2024-03-18T01:16:41.544296+0000","last_fullsized":"2024-03-18T01:16:41.544296+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:15:31.263848+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:15:31.263848+0000","last_clean_scrub_stamp":"2024-03-18T01:15:31.263848+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:33:09.523250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":5,"up_from":39,"seq":167503724559,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297644,"kb_used_data":356,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93447508,"statfs":{"total":95995035648,"available":95690248192,"internally_reserved":0,"allocated":364544,"data_stored":139193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56899999999999995}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46200000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.626}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40000000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38100000000000001}]}]},{"osd":4,"up_from":34,"seq":146028888084,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297644,"kb_used_data":356,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93447508,"statfs":{"total":95995035648,"available":95690248192,"internally_reserved":0,"allocated":364544,"data_stored":139193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 01:17:19 2024","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.83299999999999996,"5min":0.83299999999999996,"15min":0.83299999999999996},"last":0.52400000000000002},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.80900000000000005},"last":0.56399999999999995}]},{"osd":1,"last update":"Mon Mar 18 01:17:19 2024","interfaces":[{"interface":"back","average":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.66700000000000004},{"interface":"front","average":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.69699999999999995,"5min":0.69699999999999995,"15min":0.69699999999999995},"last":0.47999999999999998}]},{"osd":2,"last update":"Mon Mar 18 01:17:19 2024","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"last":0.65100000000000002},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"last":0.53700000000000003}]},{"osd":3,"last update":"Mon Mar 18 01:17:19 2024","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.626,"5min":0.626,"15min":0.626},"last":0.376},{"interface":"front","average":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"last":0.34799999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.30599999999999999}]}]},{"osd":3,"up_from":26,"seq":111669149719,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298224,"kb_used_data":936,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93446928,"statfs":{"total":95995035648,"available":95689654272,"internally_reserved":0,"allocated":958464,"data_stored":729561,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 01:16:55 2024","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.81499999999999995,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.88900000000000001,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.48799999999999999}]},{"osd":1,"last update":"Mon Mar 18 01:16:55 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.86599999999999999},"last":0.501},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.874,"5min":0.874,"15min":0.874},"last":0.53700000000000003}]},{"osd":2,"last update":"Mon Mar 18 01:16:55 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"last":0.46899999999999997}]},{"osd":4,"last update":"Mon Mar 18 01:17:17 2024","interfaces":[{"interface":"back","average":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"last":0.30599999999999999},{"interface":"front","average":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"last":0.38100000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35499999999999998}]}]},{"osd":2,"up_from":19,"seq":81604378653,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297640,"kb_used_data":352,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93447512,"statfs":{"total":95995035648,"available":95690252288,"internally_reserved":0,"allocated":360448,"data_stored":138532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 01:17:34 2024","interfaces":[{"interface":"back","average":{"1min":0.313,"5min":0.34000000000000002,"15min":0.34399999999999997},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.48399999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"last":0.30299999999999999},{"interface":"front","average":{"1min":0.313,"5min":0.34799999999999998,"15min":0.35399999999999998},"min":{"1min":0.22800000000000001,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":0.44,"5min":0.53600000000000003,"15min":0.53600000000000003},"last":0.28499999999999998}]},{"osd":1,"last update":"Mon Mar 18 01:17:34 2024","interfaces":[{"interface":"back","average":{"1min":0.317,"5min":0.36499999999999999,"15min":0.373},"min":{"1min":0.25600000000000001,"5min":0.246,"15min":0.246},"max":{"1min":0.40300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"last":0.32400000000000001},{"interface":"front","average":{"1min":0.32700000000000001,"5min":0.38200000000000001,"15min":0.39100000000000001},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.434,"5min":0.56100000000000005,"15min":0.56100000000000005},"last":0.34799999999999998}]},{"osd":3,"last update":"Mon Mar 18 01:16:56 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.59599999999999997},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"last":0.69699999999999995}]},{"osd":4,"last update":"Mon Mar 18 01:17:17 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.72799999999999998},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":0.81899999999999995,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.624}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67700000000000005}]}]},{"osd":0,"up_from":9,"seq":38654705705,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298224,"kb_used_data":936,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93446928,"statfs":{"total":95995035648,"available":95689654272,"internally_reserved":0,"allocated":958464,"data_stored":729561,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Mar 18 01:17:06 2024","interfaces":[{"interface":"back","average":{"1min":0.38800000000000001,"5min":0.32800000000000001,"15min":0.318},"min":{"1min":0.28699999999999998,"5min":0.22,"15min":0.22},"max":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"last":0.27500000000000002},{"interface":"front","average":{"1min":0.378,"5min":0.33500000000000002,"15min":0.32800000000000001},"min":{"1min":0.27800000000000002,"5min":0.214,"15min":0.214},"max":{"1min":0.49299999999999999,"5min":0.54500000000000004,"15min":0.54500000000000004},"last":0.33300000000000002}]},{"osd":2,"last update":"Mon Mar 18 01:17:31 2024","interfaces":[{"interface":"back","average":{"1min":0.38600000000000001,"5min":0.36099999999999999,"15min":0.35699999999999998},"min":{"1min":0.27300000000000002,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"last":0.35099999999999998},{"interface":"front","average":{"1min":0.39500000000000002,"5min":0.40100000000000002,"15min":0.40200000000000002},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.55900000000000005,"5min":0.57999999999999996,"15min":0.57999999999999996},"last":0.30499999999999999}]},{"osd":3,"last update":"Mon Mar 18 01:16:57 2024","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.753,"5min":0.753,"15min":0.753},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"last":0.48499999999999999}]},{"osd":4,"last update":"Mon Mar 18 01:17:18 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.55800000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.502}]}]},{"osd":1,"up_from":14,"seq":60129542178,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298224,"kb_used_data":936,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93446928,"statfs":{"total":95995035648,"available":95689654272,"internally_reserved":0,"allocated":958464,"data_stored":729561,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 01:17:02 2024","interfaces":[{"interface":"back","average":{"1min":0.36199999999999999,"5min":0.31900000000000001,"15min":0.312},"min":{"1min":0.26600000000000001,"5min":0.17599999999999999,"15min":0.17599999999999999},"max":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"last":0.35399999999999998},{"interface":"front","average":{"1min":0.36699999999999999,"5min":0.33300000000000002,"15min":0.32700000000000001},"min":{"1min":0.24099999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":0.53700000000000003,"5min":0.54300000000000004,"15min":0.54300000000000004},"last":0.42399999999999999}]},{"osd":2,"last update":"Mon Mar 18 01:17:35 2024","interfaces":[{"interface":"back","average":{"1min":0.33000000000000002,"5min":0.39000000000000001,"15min":0.40000000000000002},"min":{"1min":0.24399999999999999,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.502,"5min":0.55300000000000005,"15min":0.55300000000000005},"last":0.38800000000000001},{"interface":"front","average":{"1min":0.32900000000000001,"5min":0.39800000000000002,"15min":0.40999999999999998},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.45200000000000001,"5min":0.61199999999999999,"15min":0.61199999999999999},"last":0.37}]},{"osd":3,"last update":"Mon Mar 18 01:16:55 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"last":0.503}]},{"osd":4,"last update":"Mon Mar 18 01:17:17 2024","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.53100000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44500000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-18T01:17:39.806 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-03-18T01:17:39.806 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-03-18T01:17:39.806 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-03-18T01:17:39.806 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 37f51182-e4c4-11ee-95c9-87774f69a715 -- ceph health --format=json 2024-03-18T01:17:40.071 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/37f51182-e4c4-11ee-95c9-87774f69a715/mon.smithi029/config 2024-03-18T01:17:40.517 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:17:40 smithi029 ceph-mon[29331]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-18T01:17:40.517 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:17:40 smithi029 ceph-mon[29331]: from='client.14430 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-18T01:17:40.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:17:40 smithi090 ceph-mon[36989]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-18T01:17:40.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:17:40 smithi090 ceph-mon[36989]: from='client.14430 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-18T01:17:41.506 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:17:41 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-18T01:17:41.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:17:41 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-18T01:17:41.718 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:41.716+0000 7f0efda10700 1 -- 172.21.15.29:0/164484464 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0ef8102e90 msgr2=0x7f0ef8103260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:17:41.719 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:41.716+0000 7f0efda10700 1 --2- 172.21.15.29:0/164484464 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0ef8102e90 0x7f0ef8103260 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f0ee8004940 tx=0x7f0ee8009e30 comp rx=0 tx=0).stop 2024-03-18T01:17:41.719 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:41.716+0000 7f0efda10700 1 -- 172.21.15.29:0/164484464 shutdown_connections 2024-03-18T01:17:41.719 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:41.716+0000 7f0efda10700 1 --2- 172.21.15.29:0/164484464 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0ef80faec0 0x7f0ef80fb330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:17:41.719 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:41.716+0000 7f0efda10700 1 --2- 172.21.15.29:0/164484464 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0ef8102e90 0x7f0ef8103260 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:17:41.719 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:41.716+0000 7f0efda10700 1 -- 172.21.15.29:0/164484464 >> 172.21.15.29:0/164484464 conn(0x7f0ef80f6bd0 msgr2=0x7f0ef80f8fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:17:41.719 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:41.717+0000 7f0efda10700 1 -- 172.21.15.29:0/164484464 shutdown_connections 2024-03-18T01:17:41.719 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:41.717+0000 7f0efda10700 1 -- 172.21.15.29:0/164484464 wait complete. 2024-03-18T01:17:41.719 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:41.717+0000 7f0efda10700 1 Processor -- start 2024-03-18T01:17:41.719 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:41.717+0000 7f0efda10700 1 -- start start 2024-03-18T01:17:41.719 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:41.717+0000 7f0efda10700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0ef80faec0 0x7f0ef8113940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:17:41.719 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:41.717+0000 7f0efda10700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0ef8102e90 0x7f0ef8113e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:17:41.719 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:41.717+0000 7f0efda10700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ef81144d0 con 0x7f0ef80faec0 2024-03-18T01:17:41.719 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:41.717+0000 7f0efda10700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ef810dad0 con 0x7f0ef8102e90 2024-03-18T01:17:41.719 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:41.718+0000 7f0ef6ffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0ef80faec0 0x7f0ef8113940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:17:41.720 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:41.718+0000 7f0ef6ffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0ef80faec0 0x7f0ef8113940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39396/0 (socket says 172.21.15.29:39396) 2024-03-18T01:17:41.720 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:41.718+0000 7f0ef6ffd700 1 -- 172.21.15.29:0/1800284589 learned_addr learned my addr 172.21.15.29:0/1800284589 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-18T01:17:41.720 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:41.718+0000 7f0ef67fc700 1 --2- 172.21.15.29:0/1800284589 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0ef8102e90 0x7f0ef8113e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:17:41.720 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:41.718+0000 7f0ef6ffd700 1 -- 172.21.15.29:0/1800284589 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0ef8102e90 msgr2=0x7f0ef8113e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:17:41.720 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:41.718+0000 7f0ef6ffd700 1 --2- 172.21.15.29:0/1800284589 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0ef8102e90 0x7f0ef8113e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:17:41.720 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:41.718+0000 7f0ef6ffd700 1 -- 172.21.15.29:0/1800284589 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ee8005040 con 0x7f0ef80faec0 2024-03-18T01:17:41.720 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:41.718+0000 7f0ef6ffd700 1 --2- 172.21.15.29:0/1800284589 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0ef80faec0 0x7f0ef8113940 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f0ee8009100 tx=0x7f0ee8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:17:41.720 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:41.718+0000 7f0edf7fe700 1 -- 172.21.15.29:0/1800284589 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0ee800e040 con 0x7f0ef80faec0 2024-03-18T01:17:41.720 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:41.718+0000 7f0edf7fe700 1 -- 172.21.15.29:0/1800284589 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0ee8004290 con 0x7f0ef80faec0 2024-03-18T01:17:41.720 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:41.718+0000 7f0efda10700 1 -- 172.21.15.29:0/1800284589 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ef810dd50 con 0x7f0ef80faec0 2024-03-18T01:17:41.720 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:41.718+0000 7f0edf7fe700 1 -- 172.21.15.29:0/1800284589 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0ee801c3f0 con 0x7f0ef80faec0 2024-03-18T01:17:41.720 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:41.718+0000 7f0efda10700 1 -- 172.21.15.29:0/1800284589 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ef810e310 con 0x7f0ef80faec0 2024-03-18T01:17:41.721 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:41.719+0000 7f0efda10700 1 -- 172.21.15.29:0/1800284589 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ef81001d0 con 0x7f0ef80faec0 2024-03-18T01:17:41.721 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:41.720+0000 7f0edf7fe700 1 -- 172.21.15.29:0/1800284589 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f0ee8013790 con 0x7f0ef80faec0 2024-03-18T01:17:41.721 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:41.720+0000 7f0edf7fe700 1 --2- 172.21.15.29:0/1800284589 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f0ee00692a0 0x7f0ee006b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:17:41.721 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:41.720+0000 7f0edf7fe700 1 -- 172.21.15.29:0/1800284589 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f0ee8088140 con 0x7f0ef80faec0 2024-03-18T01:17:41.721 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:41.720+0000 7f0ef67fc700 1 --2- 172.21.15.29:0/1800284589 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f0ee00692a0 0x7f0ee006b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:17:41.722 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:41.720+0000 7f0ef67fc700 1 --2- 172.21.15.29:0/1800284589 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f0ee00692a0 0x7f0ee006b750 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f0ef810ec90 tx=0x7f0eec009ee0 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:17:41.725 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:41.725+0000 7f0edf7fe700 1 -- 172.21.15.29:0/1800284589 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175520 (secure 0 0 0) 0x7f0ee8058240 con 0x7f0ef80faec0 2024-03-18T01:17:41.961 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:41.961+0000 7f0efda10700 1 -- 172.21.15.29:0/1800284589 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f0ef810ef90 con 0x7f0ef80faec0 2024-03-18T01:17:41.961 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:41.961+0000 7f0edf7fe700 1 -- 172.21.15.29:0/1800284589 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7f0ee8013420 con 0x7f0ef80faec0 2024-03-18T01:17:41.962 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-18T01:17:41.962 INFO:teuthology.orchestra.run.smithi029.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-03-18T01:17:41.963 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:41.963+0000 7f0efda10700 1 -- 172.21.15.29:0/1800284589 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f0ee00692a0 msgr2=0x7f0ee006b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:17:41.964 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:41.963+0000 7f0efda10700 1 --2- 172.21.15.29:0/1800284589 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f0ee00692a0 0x7f0ee006b750 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f0ef810ec90 tx=0x7f0eec009ee0 comp rx=0 tx=0).stop 2024-03-18T01:17:41.964 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:41.963+0000 7f0efda10700 1 -- 172.21.15.29:0/1800284589 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0ef80faec0 msgr2=0x7f0ef8113940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:17:41.964 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:41.963+0000 7f0efda10700 1 --2- 172.21.15.29:0/1800284589 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0ef80faec0 0x7f0ef8113940 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f0ee8009100 tx=0x7f0ee8009e30 comp rx=0 tx=0).stop 2024-03-18T01:17:41.964 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:41.963+0000 7f0efda10700 1 -- 172.21.15.29:0/1800284589 shutdown_connections 2024-03-18T01:17:41.964 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:41.963+0000 7f0efda10700 1 --2- 172.21.15.29:0/1800284589 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f0ee00692a0 0x7f0ee006b750 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:17:41.964 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:41.963+0000 7f0efda10700 1 --2- 172.21.15.29:0/1800284589 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0ef80faec0 0x7f0ef8113940 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:17:41.964 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:41.963+0000 7f0efda10700 1 --2- 172.21.15.29:0/1800284589 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0ef8102e90 0x7f0ef8113e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:17:41.964 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:41.963+0000 7f0efda10700 1 -- 172.21.15.29:0/1800284589 >> 172.21.15.29:0/1800284589 conn(0x7f0ef80f6bd0 msgr2=0x7f0ef8107390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:17:41.964 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:41.963+0000 7f0efda10700 1 -- 172.21.15.29:0/1800284589 shutdown_connections 2024-03-18T01:17:41.964 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:41.963+0000 7f0efda10700 1 -- 172.21.15.29:0/1800284589 wait complete. 2024-03-18T01:17:42.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:17:42 smithi090 ceph-mon[36989]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-18T01:17:42.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:17:42 smithi090 ceph-mon[36989]: from='client.? 172.21.15.29:0/1800284589' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-18T01:17:42.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:17:42 smithi029 ceph-mon[29331]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-18T01:17:42.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:17:42 smithi029 ceph-mon[29331]: from='client.? 172.21.15.29:0/1800284589' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-18T01:17:42.603 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-03-18T01:17:42.603 INFO:tasks.cephadm:Setup complete, yielding 2024-03-18T01:17:42.603 INFO:teuthology.run_tasks:Running task print... 2024-03-18T01:17:42.612 INFO:teuthology.task.print:**** done end installing quincy cephadm ... 2024-03-18T01:17:42.612 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-18T01:17:42.661 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi029.front.sepia.ceph.com 2024-03-18T01:17:42.661 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 37f51182-e4c4-11ee-95c9-87774f69a715 -- bash -c 'ceph config set mgr mgr/cephadm/use_repo_digest true --force' 2024-03-18T01:17:42.927 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/37f51182-e4c4-11ee-95c9-87774f69a715/mon.smithi029/config 2024-03-18T01:17:44.439 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:17:44 smithi029 ceph-mon[29331]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-18T01:17:44.439 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:17:44 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T01:17:44.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:17:44 smithi090 ceph-mon[36989]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-18T01:17:44.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:17:44 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T01:17:44.547 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:44.543+0000 7f8836bf2700 1 -- 172.21.15.29:0/1229442085 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f88300fed90 msgr2=0x7f88300faed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:17:44.548 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:44.543+0000 7f8836bf2700 1 --2- 172.21.15.29:0/1229442085 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f88300fed90 0x7f88300faed0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f881c004940 tx=0x7f881c009e30 comp rx=0 tx=0).stop 2024-03-18T01:17:44.548 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:44.543+0000 7f8836bf2700 1 -- 172.21.15.29:0/1229442085 shutdown_connections 2024-03-18T01:17:44.548 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:44.543+0000 7f8836bf2700 1 --2- 172.21.15.29:0/1229442085 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f88300fed90 0x7f88300faed0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:17:44.548 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:44.543+0000 7f8836bf2700 1 --2- 172.21.15.29:0/1229442085 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f88300fe440 0x7f88300fe850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:17:44.548 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:44.543+0000 7f8836bf2700 1 -- 172.21.15.29:0/1229442085 >> 172.21.15.29:0/1229442085 conn(0x7f88300f6be0 msgr2=0x7f88300f9010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:17:44.548 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:44.543+0000 7f8836bf2700 1 -- 172.21.15.29:0/1229442085 shutdown_connections 2024-03-18T01:17:44.548 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:44.543+0000 7f8836bf2700 1 -- 172.21.15.29:0/1229442085 wait complete. 2024-03-18T01:17:44.548 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:44.544+0000 7f8836bf2700 1 Processor -- start 2024-03-18T01:17:44.548 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:44.544+0000 7f8836bf2700 1 -- start start 2024-03-18T01:17:44.548 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:44.544+0000 7f8836bf2700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f88300fe440 0x7f883010d920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:17:44.548 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:44.544+0000 7f8836bf2700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f88300fed90 0x7f883010de60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:17:44.548 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:44.544+0000 7f8836bf2700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f883010e460 con 0x7f88300fed90 2024-03-18T01:17:44.548 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:44.544+0000 7f8836bf2700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f883010e5d0 con 0x7f88300fe440 2024-03-18T01:17:44.548 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:44.544+0000 7f882ffff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f88300fed90 0x7f883010de60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:17:44.548 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:44.544+0000 7f882ffff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f88300fed90 0x7f883010de60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42230/0 (socket says 172.21.15.29:42230) 2024-03-18T01:17:44.549 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:44.544+0000 7f882ffff700 1 -- 172.21.15.29:0/4152566026 learned_addr learned my addr 172.21.15.29:0/4152566026 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-18T01:17:44.549 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:44.544+0000 7f883498e700 1 --2- 172.21.15.29:0/4152566026 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f88300fe440 0x7f883010d920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:17:44.549 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:44.545+0000 7f882ffff700 1 -- 172.21.15.29:0/4152566026 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f88300fe440 msgr2=0x7f883010d920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:17:44.549 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:44.545+0000 7f882ffff700 1 --2- 172.21.15.29:0/4152566026 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f88300fe440 0x7f883010d920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:17:44.549 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:44.545+0000 7f882ffff700 1 -- 172.21.15.29:0/4152566026 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f881c005040 con 0x7f88300fed90 2024-03-18T01:17:44.549 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:44.545+0000 7f882ffff700 1 --2- 172.21.15.29:0/4152566026 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f88300fed90 0x7f883010de60 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f881c00b3b0 tx=0x7f881c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:17:44.549 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:44.545+0000 7f882d7fa700 1 -- 172.21.15.29:0/4152566026 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f881c00e040 con 0x7f88300fed90 2024-03-18T01:17:44.549 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:44.545+0000 7f8836bf2700 1 -- 172.21.15.29:0/4152566026 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8830112fd0 con 0x7f88300fed90 2024-03-18T01:17:44.549 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:44.545+0000 7f882d7fa700 1 -- 172.21.15.29:0/4152566026 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f881c00bbf0 con 0x7f88300fed90 2024-03-18T01:17:44.549 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:44.545+0000 7f882d7fa700 1 -- 172.21.15.29:0/4152566026 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f881c01c630 con 0x7f88300fed90 2024-03-18T01:17:44.549 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:44.545+0000 7f8836bf2700 1 -- 172.21.15.29:0/4152566026 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8830113560 con 0x7f88300fed90 2024-03-18T01:17:44.549 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:44.546+0000 7f8836bf2700 1 -- 172.21.15.29:0/4152566026 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f883004c710 con 0x7f88300fed90 2024-03-18T01:17:44.549 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:44.546+0000 7f882d7fa700 1 -- 172.21.15.29:0/4152566026 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f881c00bd60 con 0x7f88300fed90 2024-03-18T01:17:44.549 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:44.547+0000 7f882d7fa700 1 --2- 172.21.15.29:0/4152566026 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f88200692a0 0x7f882006b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:17:44.549 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:44.547+0000 7f882d7fa700 1 -- 172.21.15.29:0/4152566026 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f881c087b30 con 0x7f88300fed90 2024-03-18T01:17:44.550 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:44.547+0000 7f883498e700 1 --2- 172.21.15.29:0/4152566026 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f88200692a0 0x7f882006b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:17:44.550 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:44.547+0000 7f883498e700 1 --2- 172.21.15.29:0/4152566026 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f88200692a0 0x7f882006b750 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f8824004f40 tx=0x7f882400b040 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:17:44.551 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:44.551+0000 7f882d7fa700 1 -- 172.21.15.29:0/4152566026 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175520 (secure 0 0 0) 0x7f881c057c30 con 0x7f88300fed90 2024-03-18T01:17:44.744 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:44.743+0000 7f8836bf2700 1 -- 172.21.15.29:0/4152566026 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/use_repo_digest}] v 0) v1 -- 0x7f8830113780 con 0x7f88300fed90 2024-03-18T01:17:44.748 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:44.748+0000 7f882d7fa700 1 -- 172.21.15.29:0/4152566026 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/use_repo_digest}]=0 v20)=0 v20) v1 ==== 143+0+0 (secure 0 0 0) 0x7f881c0577c0 con 0x7f88300fed90 2024-03-18T01:17:44.750 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:44.750+0000 7f8836bf2700 1 -- 172.21.15.29:0/4152566026 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f88200692a0 msgr2=0x7f882006b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:17:44.751 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:44.750+0000 7f8836bf2700 1 --2- 172.21.15.29:0/4152566026 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f88200692a0 0x7f882006b750 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f8824004f40 tx=0x7f882400b040 comp rx=0 tx=0).stop 2024-03-18T01:17:44.751 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:44.750+0000 7f8836bf2700 1 -- 172.21.15.29:0/4152566026 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f88300fed90 msgr2=0x7f883010de60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:17:44.751 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:44.750+0000 7f8836bf2700 1 --2- 172.21.15.29:0/4152566026 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f88300fed90 0x7f883010de60 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f881c00b3b0 tx=0x7f881c009e30 comp rx=0 tx=0).stop 2024-03-18T01:17:44.751 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:44.750+0000 7f8836bf2700 1 -- 172.21.15.29:0/4152566026 shutdown_connections 2024-03-18T01:17:44.751 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:44.750+0000 7f8836bf2700 1 --2- 172.21.15.29:0/4152566026 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f88200692a0 0x7f882006b750 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:17:44.751 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:44.750+0000 7f8836bf2700 1 --2- 172.21.15.29:0/4152566026 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f88300fe440 0x7f883010d920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:17:44.751 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:44.750+0000 7f8836bf2700 1 --2- 172.21.15.29:0/4152566026 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f88300fed90 0x7f883010de60 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:17:44.751 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:44.750+0000 7f8836bf2700 1 -- 172.21.15.29:0/4152566026 >> 172.21.15.29:0/4152566026 conn(0x7f88300f6be0 msgr2=0x7f88300f8240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:17:44.751 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:44.750+0000 7f8836bf2700 1 -- 172.21.15.29:0/4152566026 shutdown_connections 2024-03-18T01:17:44.751 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:44.750+0000 7f8836bf2700 1 -- 172.21.15.29:0/4152566026 wait complete. 2024-03-18T01:17:45.374 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:17:45 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:17:45.374 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:17:45 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T01:17:45.374 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:17:45 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:17:45.374 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:17:45 smithi029 ceph-mon[29331]: from='client.? 172.21.15.29:0/4152566026' entity='client.admin' 2024-03-18T01:17:45.374 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:17:45 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T01:17:45.374 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:17:45 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:17:45.374 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:17:45 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T01:17:45.374 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:17:45 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:17:45.399 INFO:teuthology.run_tasks:Running task print... 2024-03-18T01:17:45.407 INFO:teuthology.task.print:**** done cephadm.shell ceph config set mgr... 2024-03-18T01:17:45.409 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-18T01:17:45.429 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi029.front.sepia.ceph.com 2024-03-18T01:17:45.430 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 37f51182-e4c4-11ee-95c9-87774f69a715 -- bash -c 'ceph orch status' 2024-03-18T01:17:45.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:17:45 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:17:45.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:17:45 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T01:17:45.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:17:45 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:17:45.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:17:45 smithi090 ceph-mon[36989]: from='client.? 172.21.15.29:0/4152566026' entity='client.admin' 2024-03-18T01:17:45.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:17:45 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T01:17:45.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:17:45 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:17:45.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:17:45 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T01:17:45.527 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:17:45 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:17:45.690 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/37f51182-e4c4-11ee-95c9-87774f69a715/mon.smithi029/config 2024-03-18T01:17:46.494 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:17:46 smithi029 ceph-mon[29331]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-18T01:17:46.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:17:46 smithi090 ceph-mon[36989]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-18T01:17:47.405 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:47.404+0000 7f7b154a8700 1 -- 172.21.15.29:0/3690002232 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7b100fce80 msgr2=0x7f7b100fd290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:17:47.405 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:47.404+0000 7f7b154a8700 1 --2- 172.21.15.29:0/3690002232 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7b100fce80 0x7f7b100fd290 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f7b08009ce0 tx=0x7f7b08004d10 comp rx=0 tx=0).stop 2024-03-18T01:17:47.405 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:47.404+0000 7f7b154a8700 1 -- 172.21.15.29:0/3690002232 shutdown_connections 2024-03-18T01:17:47.405 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:47.404+0000 7f7b154a8700 1 --2- 172.21.15.29:0/3690002232 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7b100fdff0 0x7f7b100fe460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:17:47.405 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:47.404+0000 7f7b154a8700 1 --2- 172.21.15.29:0/3690002232 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7b100fce80 0x7f7b100fd290 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:17:47.405 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:47.404+0000 7f7b154a8700 1 -- 172.21.15.29:0/3690002232 >> 172.21.15.29:0/3690002232 conn(0x7f7b100f8580 msgr2=0x7f7b100fa990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:17:47.405 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:47.404+0000 7f7b154a8700 1 -- 172.21.15.29:0/3690002232 shutdown_connections 2024-03-18T01:17:47.405 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:47.404+0000 7f7b154a8700 1 -- 172.21.15.29:0/3690002232 wait complete. 2024-03-18T01:17:47.406 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:47.405+0000 7f7b154a8700 1 Processor -- start 2024-03-18T01:17:47.407 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:47.405+0000 7f7b154a8700 1 -- start start 2024-03-18T01:17:47.407 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:47.405+0000 7f7b154a8700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7b100fce80 0x7f7b1010fe90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:17:47.407 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:47.405+0000 7f7b154a8700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7b100fdff0 0x7f7b101103d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:17:47.407 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:47.405+0000 7f7b154a8700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b10111140 con 0x7f7b100fce80 2024-03-18T01:17:47.407 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:47.405+0000 7f7b154a8700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b101112b0 con 0x7f7b100fdff0 2024-03-18T01:17:47.407 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:47.405+0000 7f7b0effd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7b100fce80 0x7f7b1010fe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:17:47.407 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:47.405+0000 7f7b0effd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7b100fce80 0x7f7b1010fe90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42242/0 (socket says 172.21.15.29:42242) 2024-03-18T01:17:47.407 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:47.405+0000 7f7b0e7fc700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7b100fdff0 0x7f7b101103d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:17:47.407 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:47.405+0000 7f7b0effd700 1 -- 172.21.15.29:0/2885118266 learned_addr learned my addr 172.21.15.29:0/2885118266 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-18T01:17:47.407 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:47.406+0000 7f7b0effd700 1 -- 172.21.15.29:0/2885118266 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7b100fdff0 msgr2=0x7f7b101103d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:17:47.407 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:47.406+0000 7f7b0effd700 1 --2- 172.21.15.29:0/2885118266 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7b100fdff0 0x7f7b101103d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:17:47.407 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:47.406+0000 7f7b0effd700 1 -- 172.21.15.29:0/2885118266 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b08009940 con 0x7f7b100fce80 2024-03-18T01:17:47.407 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:47.406+0000 7f7b0effd700 1 --2- 172.21.15.29:0/2885118266 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7b100fce80 0x7f7b1010fe90 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f7b08005010 tx=0x7f7b08004d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:17:47.407 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:47.406+0000 7f7b077fe700 1 -- 172.21.15.29:0/2885118266 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7b0800e040 con 0x7f7b100fce80 2024-03-18T01:17:47.407 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:47.406+0000 7f7b077fe700 1 -- 172.21.15.29:0/2885118266 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7b08008e20 con 0x7f7b100fce80 2024-03-18T01:17:47.408 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:47.406+0000 7f7b154a8700 1 -- 172.21.15.29:0/2885118266 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b101114d0 con 0x7f7b100fce80 2024-03-18T01:17:47.408 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:47.406+0000 7f7b154a8700 1 -- 172.21.15.29:0/2885118266 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b10111a90 con 0x7f7b100fce80 2024-03-18T01:17:47.408 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:47.406+0000 7f7b077fe700 1 -- 172.21.15.29:0/2885118266 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7b0801c760 con 0x7f7b100fce80 2024-03-18T01:17:47.408 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:47.407+0000 7f7b154a8700 1 -- 172.21.15.29:0/2885118266 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b10101980 con 0x7f7b100fce80 2024-03-18T01:17:47.409 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:47.407+0000 7f7b077fe700 1 -- 172.21.15.29:0/2885118266 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f7b08012b50 con 0x7f7b100fce80 2024-03-18T01:17:47.409 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:47.408+0000 7f7b077fe700 1 --2- 172.21.15.29:0/2885118266 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f7afc0692b0 0x7f7afc06b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:17:47.409 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:47.408+0000 7f7b077fe700 1 -- 172.21.15.29:0/2885118266 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f7b080889b0 con 0x7f7b100fce80 2024-03-18T01:17:47.409 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:47.408+0000 7f7b0e7fc700 1 --2- 172.21.15.29:0/2885118266 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f7afc0692b0 0x7f7afc06b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:17:47.409 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:47.408+0000 7f7b0e7fc700 1 --2- 172.21.15.29:0/2885118266 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f7afc0692b0 0x7f7afc06b760 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f7af8004f60 tx=0x7f7af800b040 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:17:47.413 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:47.412+0000 7f7b077fe700 1 -- 172.21.15.29:0/2885118266 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175520 (secure 0 0 0) 0x7f7b08054fa0 con 0x7f7b100fce80 2024-03-18T01:17:47.619 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:47.619+0000 7f7b154a8700 1 -- 172.21.15.29:0/2885118266 --> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] -- mgr_command(tid 0: {"prefix": "orch status", "target": ["mon-mgr", ""]}) v1 -- 0x7f7b10105970 con 0x7f7afc0692b0 2024-03-18T01:17:47.620 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:47.620+0000 7f7b077fe700 1 -- 172.21.15.29:0/2885118266 <== mgr.14208 v2:172.21.15.29:6800/1641454714 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7f7b10105970 con 0x7f7afc0692b0 2024-03-18T01:17:47.620 INFO:teuthology.orchestra.run.smithi029.stdout:Backend: cephadm 2024-03-18T01:17:47.620 INFO:teuthology.orchestra.run.smithi029.stdout:Available: Yes 2024-03-18T01:17:47.620 INFO:teuthology.orchestra.run.smithi029.stdout:Paused: No 2024-03-18T01:17:47.622 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:47.621+0000 7f7b154a8700 1 -- 172.21.15.29:0/2885118266 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f7afc0692b0 msgr2=0x7f7afc06b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:17:47.622 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:47.621+0000 7f7b154a8700 1 --2- 172.21.15.29:0/2885118266 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f7afc0692b0 0x7f7afc06b760 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f7af8004f60 tx=0x7f7af800b040 comp rx=0 tx=0).stop 2024-03-18T01:17:47.622 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:47.621+0000 7f7b154a8700 1 -- 172.21.15.29:0/2885118266 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7b100fce80 msgr2=0x7f7b1010fe90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:17:47.622 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:47.621+0000 7f7b154a8700 1 --2- 172.21.15.29:0/2885118266 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7b100fce80 0x7f7b1010fe90 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f7b08005010 tx=0x7f7b08004d10 comp rx=0 tx=0).stop 2024-03-18T01:17:47.622 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:47.622+0000 7f7b154a8700 1 -- 172.21.15.29:0/2885118266 shutdown_connections 2024-03-18T01:17:47.622 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:47.622+0000 7f7b154a8700 1 --2- 172.21.15.29:0/2885118266 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f7afc0692b0 0x7f7afc06b760 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:17:47.623 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:47.622+0000 7f7b154a8700 1 --2- 172.21.15.29:0/2885118266 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7b100fce80 0x7f7b1010fe90 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:17:47.623 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:47.622+0000 7f7b154a8700 1 --2- 172.21.15.29:0/2885118266 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7b100fdff0 0x7f7b101103d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:17:47.623 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:47.622+0000 7f7b154a8700 1 -- 172.21.15.29:0/2885118266 >> 172.21.15.29:0/2885118266 conn(0x7f7b100f8580 msgr2=0x7f7b100f9d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:17:47.623 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:47.622+0000 7f7b154a8700 1 -- 172.21.15.29:0/2885118266 shutdown_connections 2024-03-18T01:17:47.623 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:47.622+0000 7f7b154a8700 1 -- 172.21.15.29:0/2885118266 wait complete. 2024-03-18T01:17:48.334 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 37f51182-e4c4-11ee-95c9-87774f69a715 -- bash -c 'ceph orch ps' 2024-03-18T01:17:48.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:17:48 smithi090 ceph-mon[36989]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-18T01:17:48.568 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:17:48 smithi029 ceph-mon[29331]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-18T01:17:48.597 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/37f51182-e4c4-11ee-95c9-87774f69a715/mon.smithi029/config 2024-03-18T01:17:49.319 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:17:49 smithi029 ceph-mon[29331]: from='client.14442 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T01:17:49.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:17:49 smithi090 ceph-mon[36989]: from='client.14442 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T01:17:50.293 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:50.291+0000 7ff929069700 1 -- 172.21.15.29:0/4006958804 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff9240fbb90 msgr2=0x7ff9241023a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:17:50.293 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:50.291+0000 7ff929069700 1 --2- 172.21.15.29:0/4006958804 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff9240fbb90 0x7ff9241023a0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7ff918004940 tx=0x7ff918009e30 comp rx=0 tx=0).stop 2024-03-18T01:17:50.293 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:50.291+0000 7ff929069700 1 -- 172.21.15.29:0/4006958804 shutdown_connections 2024-03-18T01:17:50.293 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:50.291+0000 7ff929069700 1 --2- 172.21.15.29:0/4006958804 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff9240fbb90 0x7ff9241023a0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:17:50.293 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:50.291+0000 7ff929069700 1 --2- 172.21.15.29:0/4006958804 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff9240fb220 0x7ff9240fb650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:17:50.293 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:50.291+0000 7ff929069700 1 -- 172.21.15.29:0/4006958804 >> 172.21.15.29:0/4006958804 conn(0x7ff9240f6be0 msgr2=0x7ff9240f9010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:17:50.293 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:50.291+0000 7ff929069700 1 -- 172.21.15.29:0/4006958804 shutdown_connections 2024-03-18T01:17:50.293 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:50.291+0000 7ff929069700 1 -- 172.21.15.29:0/4006958804 wait complete. 2024-03-18T01:17:50.293 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:50.292+0000 7ff929069700 1 Processor -- start 2024-03-18T01:17:50.293 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:50.292+0000 7ff929069700 1 -- start start 2024-03-18T01:17:50.293 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:50.292+0000 7ff929069700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff9240fb220 0x7ff9240fdb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:17:50.294 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:50.292+0000 7ff929069700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff9240fbb90 0x7ff9240fc200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:17:50.294 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:50.292+0000 7ff929069700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9240fe150 con 0x7ff9240fbb90 2024-03-18T01:17:50.294 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:50.292+0000 7ff929069700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9240fc740 con 0x7ff9240fb220 2024-03-18T01:17:50.294 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:50.292+0000 7ff92259c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff9240fbb90 0x7ff9240fc200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:17:50.294 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:50.292+0000 7ff92259c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff9240fbb90 0x7ff9240fc200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42260/0 (socket says 172.21.15.29:42260) 2024-03-18T01:17:50.294 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:50.292+0000 7ff92259c700 1 -- 172.21.15.29:0/3832324505 learned_addr learned my addr 172.21.15.29:0/3832324505 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-18T01:17:50.294 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:50.292+0000 7ff922d9d700 1 --2- 172.21.15.29:0/3832324505 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff9240fb220 0x7ff9240fdb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:17:50.294 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:50.293+0000 7ff92259c700 1 -- 172.21.15.29:0/3832324505 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff9240fb220 msgr2=0x7ff9240fdb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:17:50.294 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:50.293+0000 7ff92259c700 1 --2- 172.21.15.29:0/3832324505 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff9240fb220 0x7ff9240fdb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:17:50.294 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:50.293+0000 7ff92259c700 1 -- 172.21.15.29:0/3832324505 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff918005040 con 0x7ff9240fbb90 2024-03-18T01:17:50.294 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:50.293+0000 7ff92259c700 1 --2- 172.21.15.29:0/3832324505 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff9240fbb90 0x7ff9240fc200 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7ff918009150 tx=0x7ff918009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:17:50.295 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:50.293+0000 7ff90b7fe700 1 -- 172.21.15.29:0/3832324505 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff91800e040 con 0x7ff9240fbb90 2024-03-18T01:17:50.295 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:50.293+0000 7ff90b7fe700 1 -- 172.21.15.29:0/3832324505 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff918004290 con 0x7ff9240fbb90 2024-03-18T01:17:50.296 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:50.293+0000 7ff929069700 1 -- 172.21.15.29:0/3832324505 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9240fc990 con 0x7ff9240fbb90 2024-03-18T01:17:50.296 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:50.293+0000 7ff90b7fe700 1 -- 172.21.15.29:0/3832324505 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff91801c630 con 0x7ff9240fbb90 2024-03-18T01:17:50.296 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:50.293+0000 7ff929069700 1 -- 172.21.15.29:0/3832324505 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9240fcf50 con 0x7ff9240fbb90 2024-03-18T01:17:50.296 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:50.294+0000 7ff929069700 1 -- 172.21.15.29:0/3832324505 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff92404c710 con 0x7ff9240fbb90 2024-03-18T01:17:50.296 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:50.295+0000 7ff90b7fe700 1 -- 172.21.15.29:0/3832324505 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7ff91800bbf0 con 0x7ff9240fbb90 2024-03-18T01:17:50.296 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:50.295+0000 7ff90b7fe700 1 --2- 172.21.15.29:0/3832324505 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7ff90c0692a0 0x7ff90c06b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:17:50.296 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:50.295+0000 7ff90b7fe700 1 -- 172.21.15.29:0/3832324505 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7ff918087b20 con 0x7ff9240fbb90 2024-03-18T01:17:50.296 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:50.295+0000 7ff922d9d700 1 --2- 172.21.15.29:0/3832324505 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7ff90c0692a0 0x7ff90c06b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:17:50.296 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:50.295+0000 7ff922d9d700 1 --2- 172.21.15.29:0/3832324505 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7ff90c0692a0 0x7ff90c06b750 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7ff9240fe310 tx=0x7ff914009040 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:17:50.302 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:50.299+0000 7ff90b7fe700 1 -- 172.21.15.29:0/3832324505 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175520 (secure 0 0 0) 0x7ff918057c20 con 0x7ff9240fbb90 2024-03-18T01:17:50.318 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:17:50 smithi029 ceph-mon[29331]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-18T01:17:50.502 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:50.502+0000 7ff929069700 1 -- 172.21.15.29:0/3832324505 --> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7ff924101b80 con 0x7ff90c0692a0 2024-03-18T01:17:50.511 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:50.510+0000 7ff90b7fe700 1 -- 172.21.15.29:0/3832324505 <== mgr.14208 v2:172.21.15.29:6800/1641454714 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+2700 (secure 0 0 0) 0x7ff924101b80 con 0x7ff90c0692a0 2024-03-18T01:17:50.511 INFO:teuthology.orchestra.run.smithi029.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-18T01:17:50.511 INFO:teuthology.orchestra.run.smithi029.stdout:alertmanager.smithi029 smithi029 *:9093,9094 running (3m) 2m ago 5m 26.0M - 0.25.0 c8568f914cd2 74aa94ac819f 2024-03-18T01:17:50.511 INFO:teuthology.orchestra.run.smithi029.stdout:crash.smithi029 smithi029 running (5m) 2m ago 5m 6660k - 17.2.7-684-g6716c9f3 19857db3290c a7a7a1f5abb1 2024-03-18T01:17:50.511 INFO:teuthology.orchestra.run.smithi029.stdout:crash.smithi090 smithi090 running (4m) 70s ago 4m 6665k - 17.2.7-684-g6716c9f3 19857db3290c a0efb58cf729 2024-03-18T01:17:50.511 INFO:teuthology.orchestra.run.smithi029.stdout:grafana.smithi029 smithi029 *:3000 running (3m) 2m ago 4m 79.9M - 9.4.7 954c08fa6188 875561d97b0e 2024-03-18T01:17:50.511 INFO:teuthology.orchestra.run.smithi029.stdout:mgr.smithi029.qviwhl smithi029 *:9283 running (6m) 2m ago 6m 472M - 17.2.7-684-g6716c9f3 19857db3290c 3ff37450535a 2024-03-18T01:17:50.511 INFO:teuthology.orchestra.run.smithi029.stdout:mgr.smithi090.pnqedy smithi090 *:8443,9283 running (4m) 70s ago 4m 432M - 17.2.7-684-g6716c9f3 19857db3290c 68666c79e24d 2024-03-18T01:17:50.511 INFO:teuthology.orchestra.run.smithi029.stdout:mon.smithi029 smithi029 running (6m) 2m ago 7m 42.0M 2048M 17.2.7-684-g6716c9f3 19857db3290c 0a7fff1d6dfc 2024-03-18T01:17:50.511 INFO:teuthology.orchestra.run.smithi029.stdout:mon.smithi090 smithi090 running (4m) 70s ago 4m 40.8M 2048M 17.2.7-684-g6716c9f3 19857db3290c e23555eb8cd4 2024-03-18T01:17:50.511 INFO:teuthology.orchestra.run.smithi029.stdout:node-exporter.smithi029 smithi029 *:9100 running (5m) 2m ago 5m 16.0M - 1.5.0 0da6a335fe13 35b19f9a6014 2024-03-18T01:17:50.512 INFO:teuthology.orchestra.run.smithi029.stdout:node-exporter.smithi090 smithi090 *:9100 running (4m) 70s ago 4m 16.0M - 1.5.0 0da6a335fe13 46172a1dd8f1 2024-03-18T01:17:50.512 INFO:teuthology.orchestra.run.smithi029.stdout:osd.0 smithi029 running (3m) 2m ago 3m 38.4M 4289M 17.2.7-684-g6716c9f3 19857db3290c b34def6ff829 2024-03-18T01:17:50.512 INFO:teuthology.orchestra.run.smithi029.stdout:osd.1 smithi029 running (2m) 2m ago 2m 37.8M 4289M 17.2.7-684-g6716c9f3 19857db3290c f8915708098a 2024-03-18T01:17:50.512 INFO:teuthology.orchestra.run.smithi029.stdout:osd.2 smithi029 running (2m) 2m ago 2m 38.5M 4289M 17.2.7-684-g6716c9f3 19857db3290c cbf4647c043d 2024-03-18T01:17:50.512 INFO:teuthology.orchestra.run.smithi029.stdout:osd.3 smithi090 running (2m) 70s ago 2m 39.9M 5312M 17.2.7-684-g6716c9f3 19857db3290c dc4a91ea75c6 2024-03-18T01:17:50.512 INFO:teuthology.orchestra.run.smithi029.stdout:osd.4 smithi090 running (100s) 70s ago 99s 39.1M 5312M 17.2.7-684-g6716c9f3 19857db3290c 64754105e490 2024-03-18T01:17:50.512 INFO:teuthology.orchestra.run.smithi029.stdout:osd.5 smithi090 running (75s) 70s ago 75s 10.0M 5312M 17.2.7-684-g6716c9f3 19857db3290c 951ee2b3bc44 2024-03-18T01:17:50.512 INFO:teuthology.orchestra.run.smithi029.stdout:prometheus.smithi029 smithi029 *:9095 running (3m) 2m ago 4m 36.2M - 2.43.0 a07b618ecd1d a36de8869940 2024-03-18T01:17:50.513 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:50.513+0000 7ff929069700 1 -- 172.21.15.29:0/3832324505 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7ff90c0692a0 msgr2=0x7ff90c06b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:17:50.513 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:50.513+0000 7ff929069700 1 --2- 172.21.15.29:0/3832324505 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7ff90c0692a0 0x7ff90c06b750 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7ff9240fe310 tx=0x7ff914009040 comp rx=0 tx=0).stop 2024-03-18T01:17:50.513 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:50.513+0000 7ff929069700 1 -- 172.21.15.29:0/3832324505 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff9240fbb90 msgr2=0x7ff9240fc200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:17:50.513 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:50.513+0000 7ff929069700 1 --2- 172.21.15.29:0/3832324505 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff9240fbb90 0x7ff9240fc200 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7ff918009150 tx=0x7ff918009e30 comp rx=0 tx=0).stop 2024-03-18T01:17:50.513 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:50.513+0000 7ff929069700 1 -- 172.21.15.29:0/3832324505 shutdown_connections 2024-03-18T01:17:50.513 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:50.513+0000 7ff929069700 1 --2- 172.21.15.29:0/3832324505 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7ff90c0692a0 0x7ff90c06b750 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:17:50.513 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:50.513+0000 7ff929069700 1 --2- 172.21.15.29:0/3832324505 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff9240fb220 0x7ff9240fdb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:17:50.514 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:50.513+0000 7ff929069700 1 --2- 172.21.15.29:0/3832324505 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff9240fbb90 0x7ff9240fc200 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:17:50.514 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:50.513+0000 7ff929069700 1 -- 172.21.15.29:0/3832324505 >> 172.21.15.29:0/3832324505 conn(0x7ff9240f6be0 msgr2=0x7ff924100460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:17:50.514 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:50.513+0000 7ff929069700 1 -- 172.21.15.29:0/3832324505 shutdown_connections 2024-03-18T01:17:50.514 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:50.513+0000 7ff929069700 1 -- 172.21.15.29:0/3832324505 wait complete. 2024-03-18T01:17:50.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:17:50 smithi090 ceph-mon[36989]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-18T01:17:51.131 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 37f51182-e4c4-11ee-95c9-87774f69a715 -- bash -c 'ceph orch ls' 2024-03-18T01:17:51.394 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/37f51182-e4c4-11ee-95c9-87774f69a715/mon.smithi029/config 2024-03-18T01:17:52.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:17:52 smithi090 ceph-mon[36989]: from='client.14446 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T01:17:52.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:17:52 smithi090 ceph-mon[36989]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-18T01:17:52.526 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:17:52 smithi029 ceph-mon[29331]: from='client.14446 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T01:17:52.526 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:17:52 smithi029 ceph-mon[29331]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-18T01:17:52.930 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:52.928+0000 7f7e45cf9700 1 -- 172.21.15.29:0/1982051169 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7e400fdd90 msgr2=0x7f7e401046e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:17:52.930 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:52.928+0000 7f7e45cf9700 1 --2- 172.21.15.29:0/1982051169 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7e400fdd90 0x7f7e401046e0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f7e34004970 tx=0x7f7e34009e30 comp rx=0 tx=0).stop 2024-03-18T01:17:52.930 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:52.928+0000 7f7e45cf9700 1 -- 172.21.15.29:0/1982051169 shutdown_connections 2024-03-18T01:17:52.930 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:52.928+0000 7f7e45cf9700 1 --2- 172.21.15.29:0/1982051169 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7e400fdd90 0x7f7e401046e0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:17:52.931 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:52.928+0000 7f7e45cf9700 1 --2- 172.21.15.29:0/1982051169 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7e400fd440 0x7f7e400fd850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:17:52.931 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:52.928+0000 7f7e45cf9700 1 -- 172.21.15.29:0/1982051169 >> 172.21.15.29:0/1982051169 conn(0x7f7e400f6bd0 msgr2=0x7f7e400f9000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:17:52.931 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:52.928+0000 7f7e45cf9700 1 -- 172.21.15.29:0/1982051169 shutdown_connections 2024-03-18T01:17:52.931 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:52.928+0000 7f7e45cf9700 1 -- 172.21.15.29:0/1982051169 wait complete. 2024-03-18T01:17:52.931 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:52.929+0000 7f7e45cf9700 1 Processor -- start 2024-03-18T01:17:52.931 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:52.929+0000 7f7e45cf9700 1 -- start start 2024-03-18T01:17:52.931 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:52.929+0000 7f7e45cf9700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7e400fd440 0x7f7e4010fb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:17:52.931 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:52.929+0000 7f7e45cf9700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7e400fdd90 0x7f7e40110050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:17:52.931 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:52.929+0000 7f7e45cf9700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e401106e0 con 0x7f7e400fdd90 2024-03-18T01:17:52.931 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:52.929+0000 7f7e45cf9700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e40115080 con 0x7f7e400fd440 2024-03-18T01:17:52.931 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:52.929+0000 7f7e3effd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7e400fdd90 0x7f7e40110050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:17:52.931 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:52.929+0000 7f7e3effd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7e400fdd90 0x7f7e40110050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:48206/0 (socket says 172.21.15.29:48206) 2024-03-18T01:17:52.931 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:52.929+0000 7f7e3effd700 1 -- 172.21.15.29:0/502009242 learned_addr learned my addr 172.21.15.29:0/502009242 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-18T01:17:52.931 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:52.929+0000 7f7e3effd700 1 -- 172.21.15.29:0/502009242 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7e400fd440 msgr2=0x7f7e4010fb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:17:52.931 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:52.929+0000 7f7e3f7fe700 1 --2- 172.21.15.29:0/502009242 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7e400fd440 0x7f7e4010fb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:17:52.932 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:52.929+0000 7f7e3effd700 1 --2- 172.21.15.29:0/502009242 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7e400fd440 0x7f7e4010fb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:17:52.932 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:52.929+0000 7f7e3effd700 1 -- 172.21.15.29:0/502009242 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e34005040 con 0x7f7e400fdd90 2024-03-18T01:17:52.932 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:52.930+0000 7f7e3f7fe700 1 --2- 172.21.15.29:0/502009242 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7e400fd440 0x7f7e4010fb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T01:17:52.932 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:52.930+0000 7f7e3effd700 1 --2- 172.21.15.29:0/502009242 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7e400fdd90 0x7f7e40110050 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f7e3400b930 tx=0x7f7e34009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:17:52.932 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:52.930+0000 7f7e27fff700 1 -- 172.21.15.29:0/502009242 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7e3400e040 con 0x7f7e400fdd90 2024-03-18T01:17:52.932 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:52.930+0000 7f7e45cf9700 1 -- 172.21.15.29:0/502009242 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e40115280 con 0x7f7e400fdd90 2024-03-18T01:17:52.932 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:52.930+0000 7f7e27fff700 1 -- 172.21.15.29:0/502009242 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7e3400bce0 con 0x7f7e400fdd90 2024-03-18T01:17:52.932 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:52.930+0000 7f7e27fff700 1 -- 172.21.15.29:0/502009242 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7e3401c650 con 0x7f7e400fdd90 2024-03-18T01:17:52.932 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:52.930+0000 7f7e45cf9700 1 -- 172.21.15.29:0/502009242 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e40115770 con 0x7f7e400fdd90 2024-03-18T01:17:52.933 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:52.931+0000 7f7e45cf9700 1 -- 172.21.15.29:0/502009242 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e4004c710 con 0x7f7e400fdd90 2024-03-18T01:17:52.933 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:52.931+0000 7f7e27fff700 1 -- 172.21.15.29:0/502009242 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f7e34013a60 con 0x7f7e400fdd90 2024-03-18T01:17:52.933 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:52.932+0000 7f7e27fff700 1 --2- 172.21.15.29:0/502009242 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f7e280692a0 0x7f7e2806b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:17:52.933 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:52.932+0000 7f7e27fff700 1 -- 172.21.15.29:0/502009242 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f7e340882e0 con 0x7f7e400fdd90 2024-03-18T01:17:52.933 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:52.932+0000 7f7e3f7fe700 1 --2- 172.21.15.29:0/502009242 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f7e280692a0 0x7f7e2806b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:17:52.933 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:52.932+0000 7f7e3f7fe700 1 --2- 172.21.15.29:0/502009242 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f7e280692a0 0x7f7e2806b750 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f7e30006fd0 tx=0x7f7e30008040 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:17:52.936 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:52.936+0000 7f7e27fff700 1 -- 172.21.15.29:0/502009242 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175520 (secure 0 0 0) 0x7f7e34058360 con 0x7f7e400fdd90 2024-03-18T01:17:53.136 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:53.135+0000 7f7e45cf9700 1 -- 172.21.15.29:0/502009242 --> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] -- mgr_command(tid 0: {"prefix": "orch ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f7e400f8710 con 0x7f7e280692a0 2024-03-18T01:17:53.139 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:53.139+0000 7f7e27fff700 1 -- 172.21.15.29:0/502009242 <== mgr.14208 v2:172.21.15.29:6800/1641454714 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+1161 (secure 0 0 0) 0x7f7e400f8710 con 0x7f7e280692a0 2024-03-18T01:17:53.140 INFO:teuthology.orchestra.run.smithi029.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-03-18T01:17:53.140 INFO:teuthology.orchestra.run.smithi029.stdout:alertmanager ?:9093,9094 1/1 2m ago 6m count:1 2024-03-18T01:17:53.140 INFO:teuthology.orchestra.run.smithi029.stdout:crash 2/2 2m ago 6m * 2024-03-18T01:17:53.140 INFO:teuthology.orchestra.run.smithi029.stdout:grafana ?:3000 1/1 2m ago 6m count:1 2024-03-18T01:17:53.140 INFO:teuthology.orchestra.run.smithi029.stdout:mgr 2/2 2m ago 6m count:2 2024-03-18T01:17:53.140 INFO:teuthology.orchestra.run.smithi029.stdout:mon 2/2 2m ago 5m smithi029:172.21.15.29=smithi029;smithi090:172.21.15.90=smithi090;count:2 2024-03-18T01:17:53.140 INFO:teuthology.orchestra.run.smithi029.stdout:node-exporter ?:9100 2/2 2m ago 6m * 2024-03-18T01:17:53.140 INFO:teuthology.orchestra.run.smithi029.stdout:osd 6 2m ago - 2024-03-18T01:17:53.140 INFO:teuthology.orchestra.run.smithi029.stdout:prometheus ?:9095 1/1 2m ago 6m count:1 2024-03-18T01:17:53.142 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:53.141+0000 7f7e45cf9700 1 -- 172.21.15.29:0/502009242 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f7e280692a0 msgr2=0x7f7e2806b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:17:53.142 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:53.141+0000 7f7e45cf9700 1 --2- 172.21.15.29:0/502009242 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f7e280692a0 0x7f7e2806b750 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f7e30006fd0 tx=0x7f7e30008040 comp rx=0 tx=0).stop 2024-03-18T01:17:53.142 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:53.141+0000 7f7e45cf9700 1 -- 172.21.15.29:0/502009242 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7e400fdd90 msgr2=0x7f7e40110050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:17:53.142 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:53.141+0000 7f7e45cf9700 1 --2- 172.21.15.29:0/502009242 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7e400fdd90 0x7f7e40110050 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f7e3400b930 tx=0x7f7e34009e30 comp rx=0 tx=0).stop 2024-03-18T01:17:53.142 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:53.141+0000 7f7e45cf9700 1 -- 172.21.15.29:0/502009242 shutdown_connections 2024-03-18T01:17:53.142 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:53.141+0000 7f7e45cf9700 1 --2- 172.21.15.29:0/502009242 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f7e280692a0 0x7f7e2806b750 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:17:53.142 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:53.141+0000 7f7e45cf9700 1 --2- 172.21.15.29:0/502009242 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7e400fd440 0x7f7e4010fb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:17:53.142 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:53.141+0000 7f7e45cf9700 1 --2- 172.21.15.29:0/502009242 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7e400fdd90 0x7f7e40110050 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:17:53.142 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:53.141+0000 7f7e45cf9700 1 -- 172.21.15.29:0/502009242 >> 172.21.15.29:0/502009242 conn(0x7f7e400f6bd0 msgr2=0x7f7e400f6ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:17:53.143 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:53.141+0000 7f7e45cf9700 1 -- 172.21.15.29:0/502009242 shutdown_connections 2024-03-18T01:17:53.143 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:53.141+0000 7f7e45cf9700 1 -- 172.21.15.29:0/502009242 wait complete. 2024-03-18T01:17:53.777 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 37f51182-e4c4-11ee-95c9-87774f69a715 -- bash -c 'ceph orch host ls' 2024-03-18T01:17:54.040 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/37f51182-e4c4-11ee-95c9-87774f69a715/mon.smithi029/config 2024-03-18T01:17:54.425 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:17:54 smithi029 ceph-mon[29331]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-18T01:17:54.425 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:17:54 smithi029 ceph-mon[29331]: from='client.14450 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T01:17:54.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:17:54 smithi090 ceph-mon[36989]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-18T01:17:54.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:17:54 smithi090 ceph-mon[36989]: from='client.14450 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T01:17:55.676 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:55.673+0000 7ff410485700 1 -- 172.21.15.29:0/895597656 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff4080fff80 msgr2=0x7ff408102360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:17:55.676 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:55.673+0000 7ff410485700 1 --2- 172.21.15.29:0/895597656 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff4080fff80 0x7ff408102360 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7ff3fc004940 tx=0x7ff3fc009e30 comp rx=0 tx=0).stop 2024-03-18T01:17:55.676 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:55.673+0000 7ff410485700 1 -- 172.21.15.29:0/895597656 shutdown_connections 2024-03-18T01:17:55.677 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:55.673+0000 7ff410485700 1 --2- 172.21.15.29:0/895597656 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff4080fff80 0x7ff408102360 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:17:55.677 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:55.673+0000 7ff410485700 1 --2- 172.21.15.29:0/895597656 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff408066260 0x7ff4080ffa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:17:55.677 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:55.673+0000 7ff410485700 1 -- 172.21.15.29:0/895597656 >> 172.21.15.29:0/895597656 conn(0x7ff4080f7430 msgr2=0x7ff4080f9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:17:55.677 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:55.673+0000 7ff410485700 1 -- 172.21.15.29:0/895597656 shutdown_connections 2024-03-18T01:17:55.677 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:55.673+0000 7ff410485700 1 -- 172.21.15.29:0/895597656 wait complete. 2024-03-18T01:17:55.677 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:55.673+0000 7ff410485700 1 Processor -- start 2024-03-18T01:17:55.677 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:55.674+0000 7ff410485700 1 -- start start 2024-03-18T01:17:55.677 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:55.674+0000 7ff410485700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff408066260 0x7ff40810da30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:17:55.677 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:55.674+0000 7ff410485700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff4080fff80 0x7ff40810df70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:17:55.677 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:55.674+0000 7ff410485700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff40810e570 con 0x7ff408066260 2024-03-18T01:17:55.677 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:55.674+0000 7ff410485700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff40810edc0 con 0x7ff4080fff80 2024-03-18T01:17:55.677 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:55.674+0000 7ff40e221700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff408066260 0x7ff40810da30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:17:55.678 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:55.674+0000 7ff40e221700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff408066260 0x7ff40810da30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:48224/0 (socket says 172.21.15.29:48224) 2024-03-18T01:17:55.678 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:55.674+0000 7ff40e221700 1 -- 172.21.15.29:0/3427657386 learned_addr learned my addr 172.21.15.29:0/3427657386 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-18T01:17:55.678 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:55.674+0000 7ff40da20700 1 --2- 172.21.15.29:0/3427657386 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff4080fff80 0x7ff40810df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:17:55.678 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:55.674+0000 7ff40e221700 1 -- 172.21.15.29:0/3427657386 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff4080fff80 msgr2=0x7ff40810df70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:17:55.678 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:55.674+0000 7ff40e221700 1 --2- 172.21.15.29:0/3427657386 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff4080fff80 0x7ff40810df70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:17:55.678 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:55.674+0000 7ff40e221700 1 -- 172.21.15.29:0/3427657386 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3fc005040 con 0x7ff408066260 2024-03-18T01:17:55.678 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:55.674+0000 7ff40e221700 1 --2- 172.21.15.29:0/3427657386 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff408066260 0x7ff40810da30 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7ff400001ce0 tx=0x7ff400009e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:17:55.678 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:55.674+0000 7ff406ffd700 1 -- 172.21.15.29:0/3427657386 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff40000db60 con 0x7ff408066260 2024-03-18T01:17:55.678 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:55.674+0000 7ff406ffd700 1 -- 172.21.15.29:0/3427657386 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff40000f040 con 0x7ff408066260 2024-03-18T01:17:55.678 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:55.674+0000 7ff410485700 1 -- 172.21.15.29:0/3427657386 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff40810f0a0 con 0x7ff408066260 2024-03-18T01:17:55.678 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:55.674+0000 7ff406ffd700 1 -- 172.21.15.29:0/3427657386 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff400009790 con 0x7ff408066260 2024-03-18T01:17:55.678 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:55.674+0000 7ff410485700 1 -- 172.21.15.29:0/3427657386 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff40810f5e0 con 0x7ff408066260 2024-03-18T01:17:55.679 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:55.676+0000 7ff410485700 1 -- 172.21.15.29:0/3427657386 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff40805f5f0 con 0x7ff408066260 2024-03-18T01:17:55.679 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:55.676+0000 7ff406ffd700 1 -- 172.21.15.29:0/3427657386 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7ff400007920 con 0x7ff408066260 2024-03-18T01:17:55.680 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:55.676+0000 7ff406ffd700 1 --2- 172.21.15.29:0/3427657386 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7ff3f00692a0 0x7ff3f006b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:17:55.680 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:55.676+0000 7ff406ffd700 1 -- 172.21.15.29:0/3427657386 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7ff400086920 con 0x7ff408066260 2024-03-18T01:17:55.680 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:55.676+0000 7ff40da20700 1 --2- 172.21.15.29:0/3427657386 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7ff3f00692a0 0x7ff3f006b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:17:55.680 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:55.677+0000 7ff40da20700 1 --2- 172.21.15.29:0/3427657386 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7ff3f00692a0 0x7ff3f006b750 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7ff3fc00b3b0 tx=0x7ff3fc009e30 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:17:55.682 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:55.681+0000 7ff406ffd700 1 -- 172.21.15.29:0/3427657386 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175520 (secure 0 0 0) 0x7ff400056ad0 con 0x7ff408066260 2024-03-18T01:17:55.887 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:55.885+0000 7ff410485700 1 -- 172.21.15.29:0/3427657386 --> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""]}) v1 -- 0x7ff40810f970 con 0x7ff3f00692a0 2024-03-18T01:17:55.888 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:55.887+0000 7ff406ffd700 1 -- 172.21.15.29:0/3427657386 <== mgr.14208 v2:172.21.15.29:6800/1641454714 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+145 (secure 0 0 0) 0x7ff40810f970 con 0x7ff3f00692a0 2024-03-18T01:17:55.888 INFO:teuthology.orchestra.run.smithi029.stdout:HOST ADDR LABELS STATUS 2024-03-18T01:17:55.888 INFO:teuthology.orchestra.run.smithi029.stdout:smithi029 172.21.15.29 2024-03-18T01:17:55.888 INFO:teuthology.orchestra.run.smithi029.stdout:smithi090 172.21.15.90 2024-03-18T01:17:55.888 INFO:teuthology.orchestra.run.smithi029.stdout:2 hosts in cluster 2024-03-18T01:17:55.890 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:55.889+0000 7ff410485700 1 -- 172.21.15.29:0/3427657386 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7ff3f00692a0 msgr2=0x7ff3f006b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:17:55.890 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:55.889+0000 7ff410485700 1 --2- 172.21.15.29:0/3427657386 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7ff3f00692a0 0x7ff3f006b750 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7ff3fc00b3b0 tx=0x7ff3fc009e30 comp rx=0 tx=0).stop 2024-03-18T01:17:55.890 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:55.889+0000 7ff410485700 1 -- 172.21.15.29:0/3427657386 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff408066260 msgr2=0x7ff40810da30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:17:55.890 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:55.889+0000 7ff410485700 1 --2- 172.21.15.29:0/3427657386 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff408066260 0x7ff40810da30 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7ff400001ce0 tx=0x7ff400009e60 comp rx=0 tx=0).stop 2024-03-18T01:17:55.890 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:55.889+0000 7ff410485700 1 -- 172.21.15.29:0/3427657386 shutdown_connections 2024-03-18T01:17:55.890 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:55.889+0000 7ff410485700 1 --2- 172.21.15.29:0/3427657386 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7ff3f00692a0 0x7ff3f006b750 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:17:55.890 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:55.889+0000 7ff410485700 1 --2- 172.21.15.29:0/3427657386 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff408066260 0x7ff40810da30 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:17:55.891 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:55.889+0000 7ff410485700 1 --2- 172.21.15.29:0/3427657386 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff4080fff80 0x7ff40810df70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:17:55.891 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:55.889+0000 7ff410485700 1 -- 172.21.15.29:0/3427657386 >> 172.21.15.29:0/3427657386 conn(0x7ff4080f7430 msgr2=0x7ff4080f7850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:17:55.891 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:55.889+0000 7ff410485700 1 -- 172.21.15.29:0/3427657386 shutdown_connections 2024-03-18T01:17:55.891 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:55.889+0000 7ff410485700 1 -- 172.21.15.29:0/3427657386 wait complete. 2024-03-18T01:17:56.483 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:17:56 smithi029 ceph-mon[29331]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-18T01:17:56.483 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:17:56 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-18T01:17:56.510 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 37f51182-e4c4-11ee-95c9-87774f69a715 -- bash -c 'ceph orch device ls' 2024-03-18T01:17:56.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:17:56 smithi090 ceph-mon[36989]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-18T01:17:56.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:17:56 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-18T01:17:56.772 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/37f51182-e4c4-11ee-95c9-87774f69a715/mon.smithi029/config 2024-03-18T01:17:57.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:17:57 smithi090 ceph-mon[36989]: from='client.14454 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T01:17:57.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:17:57 smithi029 ceph-mon[29331]: from='client.14454 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T01:17:58.371 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:58.364+0000 7facbe5c7700 1 -- 172.21.15.29:0/2127183973 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7facb8066280 msgr2=0x7facb80ffa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:17:58.371 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:58.364+0000 7facbe5c7700 1 --2- 172.21.15.29:0/2127183973 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7facb8066280 0x7facb80ffa20 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7faca8004940 tx=0x7faca8009e30 comp rx=0 tx=0).stop 2024-03-18T01:17:58.371 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:58.364+0000 7facbe5c7700 1 -- 172.21.15.29:0/2127183973 shutdown_connections 2024-03-18T01:17:58.371 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:58.364+0000 7facbe5c7700 1 --2- 172.21.15.29:0/2127183973 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7facb80fff60 0x7facb8102340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:17:58.372 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:58.364+0000 7facbe5c7700 1 --2- 172.21.15.29:0/2127183973 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7facb8066280 0x7facb80ffa20 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:17:58.372 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:58.364+0000 7facbe5c7700 1 -- 172.21.15.29:0/2127183973 >> 172.21.15.29:0/2127183973 conn(0x7facb80f7430 msgr2=0x7facb80f9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:17:58.372 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:58.364+0000 7facbe5c7700 1 -- 172.21.15.29:0/2127183973 shutdown_connections 2024-03-18T01:17:58.372 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:58.364+0000 7facbe5c7700 1 -- 172.21.15.29:0/2127183973 wait complete. 2024-03-18T01:17:58.372 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:58.365+0000 7facbe5c7700 1 Processor -- start 2024-03-18T01:17:58.372 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:58.365+0000 7facbe5c7700 1 -- start start 2024-03-18T01:17:58.372 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:58.365+0000 7facbe5c7700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7facb8066280 0x7facb810daa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:17:58.372 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:58.365+0000 7facbe5c7700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7facb80fff60 0x7facb810dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:17:58.372 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:58.365+0000 7facbe5c7700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facb810e5e0 con 0x7facb80fff60 2024-03-18T01:17:58.372 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:58.365+0000 7facbe5c7700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facb810ee30 con 0x7facb8066280 2024-03-18T01:17:58.372 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:58.365+0000 7facb77fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7facb80fff60 0x7facb810dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:17:58.372 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:58.365+0000 7facb77fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7facb80fff60 0x7facb810dfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:48238/0 (socket says 172.21.15.29:48238) 2024-03-18T01:17:58.372 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:58.365+0000 7facb77fe700 1 -- 172.21.15.29:0/3716698576 learned_addr learned my addr 172.21.15.29:0/3716698576 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-18T01:17:58.372 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:58.365+0000 7facb7fff700 1 --2- 172.21.15.29:0/3716698576 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7facb8066280 0x7facb810daa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:17:58.373 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:58.365+0000 7facb77fe700 1 -- 172.21.15.29:0/3716698576 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7facb8066280 msgr2=0x7facb810daa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:17:58.373 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:58.365+0000 7facb77fe700 1 --2- 172.21.15.29:0/3716698576 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7facb8066280 0x7facb810daa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:17:58.373 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:58.365+0000 7facb77fe700 1 -- 172.21.15.29:0/3716698576 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faca8005040 con 0x7facb80fff60 2024-03-18T01:17:58.373 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:58.365+0000 7facb77fe700 1 --2- 172.21.15.29:0/3716698576 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7facb80fff60 0x7facb810dfe0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7facac00c830 tx=0x7facac004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:17:58.373 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:58.366+0000 7facb4ff9700 1 -- 172.21.15.29:0/3716698576 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7facac00a760 con 0x7facb80fff60 2024-03-18T01:17:58.373 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:58.366+0000 7facb4ff9700 1 -- 172.21.15.29:0/3716698576 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7facac00ad90 con 0x7facb80fff60 2024-03-18T01:17:58.373 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:58.366+0000 7facbe5c7700 1 -- 172.21.15.29:0/3716698576 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7facb810f110 con 0x7facb80fff60 2024-03-18T01:17:58.373 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:58.366+0000 7facb4ff9700 1 -- 172.21.15.29:0/3716698576 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7facac0113a0 con 0x7facb80fff60 2024-03-18T01:17:58.373 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:58.366+0000 7facbe5c7700 1 -- 172.21.15.29:0/3716698576 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7facb810f650 con 0x7facb80fff60 2024-03-18T01:17:58.373 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:58.367+0000 7facbe5c7700 1 -- 172.21.15.29:0/3716698576 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7facb805f5f0 con 0x7facb80fff60 2024-03-18T01:17:58.374 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:58.367+0000 7facb4ff9700 1 -- 172.21.15.29:0/3716698576 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7facac012400 con 0x7facb80fff60 2024-03-18T01:17:58.374 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:58.367+0000 7facb4ff9700 1 --2- 172.21.15.29:0/3716698576 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7faca00692b0 0x7faca006b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:17:58.374 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:58.368+0000 7facb4ff9700 1 -- 172.21.15.29:0/3716698576 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7facac086e00 con 0x7facb80fff60 2024-03-18T01:17:58.374 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:58.368+0000 7facb7fff700 1 --2- 172.21.15.29:0/3716698576 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7faca00692b0 0x7faca006b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:17:58.374 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:58.368+0000 7facb7fff700 1 --2- 172.21.15.29:0/3716698576 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7faca00692b0 0x7faca006b760 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7faca800b970 tx=0x7faca8009e30 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:17:58.374 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:58.372+0000 7facb4ff9700 1 -- 172.21.15.29:0/3716698576 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175520 (secure 0 0 0) 0x7facac056fa0 con 0x7facb80fff60 2024-03-18T01:17:58.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:17:58 smithi090 ceph-mon[36989]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-18T01:17:58.568 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:17:58 smithi029 ceph-mon[29331]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-18T01:17:58.574 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:58.573+0000 7facbe5c7700 1 -- 172.21.15.29:0/3716698576 --> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] -- mgr_command(tid 0: {"prefix": "orch device ls", "target": ["mon-mgr", ""]}) v1 -- 0x7facb810fa30 con 0x7faca00692b0 2024-03-18T01:17:58.576 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:58.575+0000 7facb4ff9700 1 -- 172.21.15.29:0/3716698576 <== mgr.14208 v2:172.21.15.29:6800/1641454714 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+399 (secure 0 0 0) 0x7facb810fa30 con 0x7faca00692b0 2024-03-18T01:17:58.576 INFO:teuthology.orchestra.run.smithi029.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2024-03-18T01:17:58.576 INFO:teuthology.orchestra.run.smithi029.stdout:smithi029 /dev/nvme0n1 ssd INTEL_SSDPED1D480GA_PHMB7512009B480DGN 447G No 2m ago Has a FileSystem, LVM detected 2024-03-18T01:17:58.576 INFO:teuthology.orchestra.run.smithi029.stdout:smithi090 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT620400UK400BGN 372G No 75s ago Has a FileSystem, LVM detected 2024-03-18T01:17:58.578 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:58.576+0000 7facbe5c7700 1 -- 172.21.15.29:0/3716698576 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7faca00692b0 msgr2=0x7faca006b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:17:58.578 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:58.577+0000 7facbe5c7700 1 --2- 172.21.15.29:0/3716698576 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7faca00692b0 0x7faca006b760 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7faca800b970 tx=0x7faca8009e30 comp rx=0 tx=0).stop 2024-03-18T01:17:58.578 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:58.577+0000 7facbe5c7700 1 -- 172.21.15.29:0/3716698576 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7facb80fff60 msgr2=0x7facb810dfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:17:58.578 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:58.577+0000 7facbe5c7700 1 --2- 172.21.15.29:0/3716698576 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7facb80fff60 0x7facb810dfe0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7facac00c830 tx=0x7facac004470 comp rx=0 tx=0).stop 2024-03-18T01:17:58.578 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:58.577+0000 7facbe5c7700 1 -- 172.21.15.29:0/3716698576 shutdown_connections 2024-03-18T01:17:58.578 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:58.577+0000 7facbe5c7700 1 --2- 172.21.15.29:0/3716698576 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7faca00692b0 0x7faca006b760 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:17:58.578 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:58.577+0000 7facbe5c7700 1 --2- 172.21.15.29:0/3716698576 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7facb8066280 0x7facb810daa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:17:58.578 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:58.577+0000 7facbe5c7700 1 --2- 172.21.15.29:0/3716698576 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7facb80fff60 0x7facb810dfe0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:17:58.578 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:58.577+0000 7facbe5c7700 1 -- 172.21.15.29:0/3716698576 >> 172.21.15.29:0/3716698576 conn(0x7facb80f7430 msgr2=0x7facb80f7850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:17:58.579 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:58.577+0000 7facbe5c7700 1 -- 172.21.15.29:0/3716698576 shutdown_connections 2024-03-18T01:17:58.579 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:17:58.577+0000 7facbe5c7700 1 -- 172.21.15.29:0/3716698576 wait complete. 2024-03-18T01:17:59.323 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-18T01:17:59.332 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi029.front.sepia.ceph.com 2024-03-18T01:17:59.333 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 37f51182-e4c4-11ee-95c9-87774f69a715 -- bash -c 'ceph fs volume create cephfs --placement=4' 2024-03-18T01:17:59.594 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/37f51182-e4c4-11ee-95c9-87774f69a715/mon.smithi029/config 2024-03-18T01:18:00.319 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:18:00 smithi029 ceph-mon[29331]: from='client.14458 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T01:18:00.319 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:18:00 smithi029 ceph-mon[29331]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-18T01:18:00.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:18:00 smithi090 ceph-mon[36989]: from='client.14458 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T01:18:00.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:18:00 smithi090 ceph-mon[36989]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-18T01:18:01.251 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:01.249+0000 7feaedb5f700 1 -- 172.21.15.29:0/3007516738 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feae8066f70 msgr2=0x7feae80ff9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:18:01.251 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:01.249+0000 7feaedb5f700 1 --2- 172.21.15.29:0/3007516738 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feae8066f70 0x7feae80ff9c0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fead8004940 tx=0x7fead8009e30 comp rx=0 tx=0).stop 2024-03-18T01:18:01.251 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:01.249+0000 7feaedb5f700 1 -- 172.21.15.29:0/3007516738 shutdown_connections 2024-03-18T01:18:01.251 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:01.249+0000 7feaedb5f700 1 --2- 172.21.15.29:0/3007516738 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7feae80fff00 0x7feae8102320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:18:01.251 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:01.249+0000 7feaedb5f700 1 --2- 172.21.15.29:0/3007516738 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feae8066f70 0x7feae80ff9c0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:18:01.251 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:01.249+0000 7feaedb5f700 1 -- 172.21.15.29:0/3007516738 >> 172.21.15.29:0/3007516738 conn(0x7feae80f7430 msgr2=0x7feae80f9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:18:01.251 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:01.249+0000 7feaedb5f700 1 -- 172.21.15.29:0/3007516738 shutdown_connections 2024-03-18T01:18:01.252 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:01.249+0000 7feaedb5f700 1 -- 172.21.15.29:0/3007516738 wait complete. 2024-03-18T01:18:01.252 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:01.249+0000 7feaedb5f700 1 Processor -- start 2024-03-18T01:18:01.252 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:01.249+0000 7feaedb5f700 1 -- start start 2024-03-18T01:18:01.252 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:01.250+0000 7feaedb5f700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feae8066f70 0x7feae80fd210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:18:01.252 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:01.250+0000 7feaedb5f700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7feae80fff00 0x7feae80fb8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:18:01.252 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:01.250+0000 7feaedb5f700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feae80fd810 con 0x7feae8066f70 2024-03-18T01:18:01.252 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:01.250+0000 7feaedb5f700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feae80fbe00 con 0x7feae80fff00 2024-03-18T01:18:01.252 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:01.250+0000 7feae77fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feae8066f70 0x7feae80fd210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:18:01.252 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:01.250+0000 7feae77fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feae8066f70 0x7feae80fd210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:48254/0 (socket says 172.21.15.29:48254) 2024-03-18T01:18:01.252 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:01.250+0000 7feae77fe700 1 -- 172.21.15.29:0/2721703844 learned_addr learned my addr 172.21.15.29:0/2721703844 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-18T01:18:01.252 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:01.250+0000 7feae6ffd700 1 --2- 172.21.15.29:0/2721703844 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7feae80fff00 0x7feae80fb8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:18:01.252 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:01.250+0000 7feae77fe700 1 -- 172.21.15.29:0/2721703844 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7feae80fff00 msgr2=0x7feae80fb8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:18:01.252 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:01.250+0000 7feae77fe700 1 --2- 172.21.15.29:0/2721703844 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7feae80fff00 0x7feae80fb8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:18:01.252 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:01.250+0000 7feae77fe700 1 -- 172.21.15.29:0/2721703844 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fead8005040 con 0x7feae8066f70 2024-03-18T01:18:01.254 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:01.250+0000 7feae77fe700 1 --2- 172.21.15.29:0/2721703844 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feae8066f70 0x7feae80fd210 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fead800b3b0 tx=0x7fead8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:18:01.254 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:01.250+0000 7feacffff700 1 -- 172.21.15.29:0/2721703844 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fead800e040 con 0x7feae8066f70 2024-03-18T01:18:01.254 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:01.250+0000 7feacffff700 1 -- 172.21.15.29:0/2721703844 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fead800bbf0 con 0x7feae8066f70 2024-03-18T01:18:01.254 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:01.250+0000 7feaedb5f700 1 -- 172.21.15.29:0/2721703844 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feae80fc050 con 0x7feae8066f70 2024-03-18T01:18:01.254 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:01.250+0000 7feacffff700 1 -- 172.21.15.29:0/2721703844 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fead801c630 con 0x7feae8066f70 2024-03-18T01:18:01.254 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:01.250+0000 7feaedb5f700 1 -- 172.21.15.29:0/2721703844 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feae80fc610 con 0x7feae8066f70 2024-03-18T01:18:01.254 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:01.251+0000 7feaedb5f700 1 -- 172.21.15.29:0/2721703844 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feae804c710 con 0x7feae8066f70 2024-03-18T01:18:01.254 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:01.252+0000 7feacffff700 1 -- 172.21.15.29:0/2721703844 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7fead800bd60 con 0x7feae8066f70 2024-03-18T01:18:01.254 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:01.252+0000 7feacffff700 1 --2- 172.21.15.29:0/2721703844 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7fead00692a0 0x7fead006b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:18:01.254 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:01.252+0000 7feacffff700 1 -- 172.21.15.29:0/2721703844 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fead8087af0 con 0x7feae8066f70 2024-03-18T01:18:01.254 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:01.252+0000 7feae6ffd700 1 --2- 172.21.15.29:0/2721703844 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7fead00692a0 0x7fead006b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:18:01.254 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:01.253+0000 7feae6ffd700 1 --2- 172.21.15.29:0/2721703844 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7fead00692a0 0x7fead006b750 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7feadc0041d0 tx=0x7feadc0043c0 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:18:01.258 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:01.257+0000 7feacffff700 1 -- 172.21.15.29:0/2721703844 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175520 (secure 0 0 0) 0x7fead8057bf0 con 0x7feae8066f70 2024-03-18T01:18:01.478 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:01.476+0000 7feaedb5f700 1 -- 172.21.15.29:0/2721703844 --> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] -- mgr_command(tid 0: {"prefix": "fs volume create", "name": "cephfs", "placement": "4", "target": ["mon-mgr", ""]}) v1 -- 0x7feae80f8f70 con 0x7fead00692a0 2024-03-18T01:18:02.776 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:18:02 smithi090 ceph-mon[36989]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-18T01:18:02.776 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:18:02 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "osd pool create", "pool": "cephfs.cephfs.meta"}]: dispatch 2024-03-18T01:18:02.819 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:18:02 smithi029 ceph-mon[29331]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-18T01:18:02.819 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:18:02 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "osd pool create", "pool": "cephfs.cephfs.meta"}]: dispatch 2024-03-18T01:18:03.251 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:03.249+0000 7feacffff700 1 -- 172.21.15.29:0/2721703844 <== mgr.14208 v2:172.21.15.29:6800/1641454714 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7feae80f8f70 con 0x7fead00692a0 2024-03-18T01:18:03.254 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:03.252+0000 7feaedb5f700 1 -- 172.21.15.29:0/2721703844 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7fead00692a0 msgr2=0x7fead006b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:18:03.254 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:03.252+0000 7feaedb5f700 1 --2- 172.21.15.29:0/2721703844 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7fead00692a0 0x7fead006b750 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7feadc0041d0 tx=0x7feadc0043c0 comp rx=0 tx=0).stop 2024-03-18T01:18:03.254 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:03.252+0000 7feaedb5f700 1 -- 172.21.15.29:0/2721703844 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feae8066f70 msgr2=0x7feae80fd210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:18:03.254 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:03.252+0000 7feaedb5f700 1 --2- 172.21.15.29:0/2721703844 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feae8066f70 0x7feae80fd210 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fead800b3b0 tx=0x7fead8009e30 comp rx=0 tx=0).stop 2024-03-18T01:18:03.254 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:03.252+0000 7feaedb5f700 1 -- 172.21.15.29:0/2721703844 shutdown_connections 2024-03-18T01:18:03.254 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:03.252+0000 7feaedb5f700 1 --2- 172.21.15.29:0/2721703844 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7fead00692a0 0x7fead006b750 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:18:03.254 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:03.252+0000 7feaedb5f700 1 --2- 172.21.15.29:0/2721703844 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feae8066f70 0x7feae80fd210 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:18:03.254 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:03.252+0000 7feaedb5f700 1 --2- 172.21.15.29:0/2721703844 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7feae80fff00 0x7feae80fb8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:18:03.254 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:03.252+0000 7feaedb5f700 1 -- 172.21.15.29:0/2721703844 >> 172.21.15.29:0/2721703844 conn(0x7feae80f7430 msgr2=0x7feae80f7850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:18:03.254 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:03.252+0000 7feaedb5f700 1 -- 172.21.15.29:0/2721703844 shutdown_connections 2024-03-18T01:18:03.255 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:03.252+0000 7feaedb5f700 1 -- 172.21.15.29:0/2721703844 wait complete. 2024-03-18T01:18:03.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:18:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:18:03.226+0000 7f4cb228d700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-18T01:18:03.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:18:03 smithi029 ceph-mon[29331]: from='client.14462 -' entity='client.admin' cmd=[{"prefix": "fs volume create", "name": "cephfs", "placement": "4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T01:18:03.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:18:03 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd='[{"prefix": "osd pool create", "pool": "cephfs.cephfs.meta"}]': finished 2024-03-18T01:18:03.570 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:18:03 smithi029 ceph-mon[29331]: osdmap e41: 6 total, 6 up, 6 in 2024-03-18T01:18:03.570 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:18:03 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.cephfs.data"}]: dispatch 2024-03-18T01:18:03.570 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:18:03 smithi029 ceph-mon[29331]: pgmap v175: 33 pgs: 5 creating+peering, 1 active+clean, 27 unknown; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-18T01:18:03.570 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:18:03 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd='[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.cephfs.data"}]': finished 2024-03-18T01:18:03.570 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:18:03 smithi029 ceph-mon[29331]: osdmap e42: 6 total, 6 up, 6 in 2024-03-18T01:18:03.570 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:18:03 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs.cephfs.meta", "data": "cephfs.cephfs.data"}]: dispatch 2024-03-18T01:18:03.570 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:18:03 smithi029 ceph-mon[29331]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-18T01:18:03.570 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:18:03 smithi029 ceph-mon[29331]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-03-18T01:18:03.570 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:18:03 smithi029 ceph-mon[29331]: osdmap e43: 6 total, 6 up, 6 in 2024-03-18T01:18:03.570 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:18:03 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs.cephfs.meta", "data": "cephfs.cephfs.data"}]': finished 2024-03-18T01:18:03.570 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:18:03 smithi029 ceph-mon[29331]: fsmap cephfs:0 2024-03-18T01:18:03.570 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:18:03 smithi029 ceph-mon[29331]: Saving service mds.cephfs spec with placement count:4 2024-03-18T01:18:03.570 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:18:03 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:18:03.571 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:18:03 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T01:18:03.571 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:18:03 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:18:03.571 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:18:03 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T01:18:03.571 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:18:03 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:18:03.571 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:18:03 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi029.xuxvlr", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T01:18:03.571 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:18:03 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi029.xuxvlr", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T01:18:03.571 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:18:03 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:18:03.571 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:18:03 smithi029 ceph-mon[29331]: Deploying daemon mds.cephfs.smithi029.xuxvlr on smithi029 2024-03-18T01:18:03.776 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:18:03 smithi090 ceph-mon[36989]: from='client.14462 -' entity='client.admin' cmd=[{"prefix": "fs volume create", "name": "cephfs", "placement": "4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T01:18:03.776 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:18:03 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd='[{"prefix": "osd pool create", "pool": "cephfs.cephfs.meta"}]': finished 2024-03-18T01:18:03.776 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:18:03 smithi090 ceph-mon[36989]: osdmap e41: 6 total, 6 up, 6 in 2024-03-18T01:18:03.776 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:18:03 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.cephfs.data"}]: dispatch 2024-03-18T01:18:03.777 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:18:03 smithi090 ceph-mon[36989]: pgmap v175: 33 pgs: 5 creating+peering, 1 active+clean, 27 unknown; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-18T01:18:03.777 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:18:03 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd='[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.cephfs.data"}]': finished 2024-03-18T01:18:03.777 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:18:03 smithi090 ceph-mon[36989]: osdmap e42: 6 total, 6 up, 6 in 2024-03-18T01:18:03.777 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:18:03 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs.cephfs.meta", "data": "cephfs.cephfs.data"}]: dispatch 2024-03-18T01:18:03.777 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:18:03 smithi090 ceph-mon[36989]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-18T01:18:03.777 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:18:03 smithi090 ceph-mon[36989]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-03-18T01:18:03.777 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:18:03 smithi090 ceph-mon[36989]: osdmap e43: 6 total, 6 up, 6 in 2024-03-18T01:18:03.777 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:18:03 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs.cephfs.meta", "data": "cephfs.cephfs.data"}]': finished 2024-03-18T01:18:03.777 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:18:03 smithi090 ceph-mon[36989]: fsmap cephfs:0 2024-03-18T01:18:03.777 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:18:03 smithi090 ceph-mon[36989]: Saving service mds.cephfs spec with placement count:4 2024-03-18T01:18:03.777 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:18:03 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:18:03.777 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:18:03 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T01:18:03.777 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:18:03 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:18:03.778 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:18:03 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T01:18:03.778 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:18:03 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:18:03.778 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:18:03 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi029.xuxvlr", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T01:18:03.778 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:18:03 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi029.xuxvlr", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T01:18:03.778 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:18:03 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:18:03.778 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:18:03 smithi090 ceph-mon[36989]: Deploying daemon mds.cephfs.smithi029.xuxvlr on smithi029 2024-03-18T01:18:04.031 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 37f51182-e4c4-11ee-95c9-87774f69a715 -- bash -c 'ceph fs dump' 2024-03-18T01:18:04.776 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:18:04 smithi090 ceph-mon[36989]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-18T01:18:04.776 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:18:04 smithi090 ceph-mon[36989]: osdmap e44: 6 total, 6 up, 6 in 2024-03-18T01:18:04.819 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:18:04 smithi029 ceph-mon[29331]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-18T01:18:04.819 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:18:04 smithi029 ceph-mon[29331]: osdmap e44: 6 total, 6 up, 6 in 2024-03-18T01:18:05.220 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/37f51182-e4c4-11ee-95c9-87774f69a715/mon.smithi029/config 2024-03-18T01:18:06.286 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:18:06 smithi029 ceph-mon[29331]: pgmap v179: 65 pgs: 10 active+clean, 17 creating+peering, 38 unknown; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-18T01:18:06.286 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:18:06 smithi029 ceph-mon[29331]: osdmap e45: 6 total, 6 up, 6 in 2024-03-18T01:18:06.286 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:18:06 smithi029 ceph-mon[29331]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-18T01:18:06.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:18:06 smithi090 ceph-mon[36989]: pgmap v179: 65 pgs: 10 active+clean, 17 creating+peering, 38 unknown; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-18T01:18:06.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:18:06 smithi090 ceph-mon[36989]: osdmap e45: 6 total, 6 up, 6 in 2024-03-18T01:18:06.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:18:06 smithi090 ceph-mon[36989]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-18T01:18:07.048 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:07.041+0000 7fba45b57700 1 -- 172.21.15.29:0/4204797876 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fba400fed90 msgr2=0x7fba400faed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:18:07.049 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:07.041+0000 7fba45b57700 1 --2- 172.21.15.29:0/4204797876 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fba400fed90 0x7fba400faed0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fba34004940 tx=0x7fba34009e30 comp rx=0 tx=0).stop 2024-03-18T01:18:07.049 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:07.041+0000 7fba45b57700 1 -- 172.21.15.29:0/4204797876 shutdown_connections 2024-03-18T01:18:07.049 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:07.041+0000 7fba45b57700 1 --2- 172.21.15.29:0/4204797876 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fba400fed90 0x7fba400faed0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:18:07.049 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:07.041+0000 7fba45b57700 1 --2- 172.21.15.29:0/4204797876 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fba400fe440 0x7fba400fe850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:18:07.049 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:07.041+0000 7fba45b57700 1 -- 172.21.15.29:0/4204797876 >> 172.21.15.29:0/4204797876 conn(0x7fba400f6bc0 msgr2=0x7fba400f9010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:18:07.049 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:07.041+0000 7fba45b57700 1 -- 172.21.15.29:0/4204797876 shutdown_connections 2024-03-18T01:18:07.049 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:07.041+0000 7fba45b57700 1 -- 172.21.15.29:0/4204797876 wait complete. 2024-03-18T01:18:07.049 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:07.042+0000 7fba45b57700 1 Processor -- start 2024-03-18T01:18:07.049 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:07.042+0000 7fba45b57700 1 -- start start 2024-03-18T01:18:07.049 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:07.042+0000 7fba45b57700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fba400fe440 0x7fba4010d900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:18:07.049 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:07.042+0000 7fba45b57700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fba400fed90 0x7fba4010de40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:18:07.049 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:07.042+0000 7fba45b57700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba4010e4d0 con 0x7fba400fe440 2024-03-18T01:18:07.050 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:07.042+0000 7fba45b57700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba40112e70 con 0x7fba400fed90 2024-03-18T01:18:07.050 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:07.042+0000 7fba3f7fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fba400fe440 0x7fba4010d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:18:07.050 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:07.042+0000 7fba3f7fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fba400fe440 0x7fba4010d900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42662/0 (socket says 172.21.15.29:42662) 2024-03-18T01:18:07.050 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:07.042+0000 7fba3f7fe700 1 -- 172.21.15.29:0/3421275186 learned_addr learned my addr 172.21.15.29:0/3421275186 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-18T01:18:07.050 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:07.042+0000 7fba3effd700 1 --2- 172.21.15.29:0/3421275186 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fba400fed90 0x7fba4010de40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:18:07.050 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:07.043+0000 7fba3f7fe700 1 -- 172.21.15.29:0/3421275186 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fba400fed90 msgr2=0x7fba4010de40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:18:07.050 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:07.043+0000 7fba3f7fe700 1 --2- 172.21.15.29:0/3421275186 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fba400fed90 0x7fba4010de40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:18:07.050 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:07.043+0000 7fba3f7fe700 1 -- 172.21.15.29:0/3421275186 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba34005040 con 0x7fba400fe440 2024-03-18T01:18:07.050 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:07.043+0000 7fba3f7fe700 1 --2- 172.21.15.29:0/3421275186 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fba400fe440 0x7fba4010d900 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fba3000b790 tx=0x7fba30009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:18:07.050 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:07.043+0000 7fba27fff700 1 -- 172.21.15.29:0/3421275186 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fba3000f6f0 con 0x7fba400fe440 2024-03-18T01:18:07.050 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:07.043+0000 7fba27fff700 1 -- 172.21.15.29:0/3421275186 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fba3000fd20 con 0x7fba400fe440 2024-03-18T01:18:07.050 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:07.043+0000 7fba45b57700 1 -- 172.21.15.29:0/3421275186 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba401130d0 con 0x7fba400fe440 2024-03-18T01:18:07.050 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:07.043+0000 7fba27fff700 1 -- 172.21.15.29:0/3421275186 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fba300103a0 con 0x7fba400fe440 2024-03-18T01:18:07.051 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:07.043+0000 7fba45b57700 1 -- 172.21.15.29:0/3421275186 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba40113640 con 0x7fba400fe440 2024-03-18T01:18:07.051 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:07.044+0000 7fba45b57700 1 -- 172.21.15.29:0/3421275186 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba4004c710 con 0x7fba400fe440 2024-03-18T01:18:07.051 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:07.045+0000 7fba27fff700 1 -- 172.21.15.29:0/3421275186 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7fba30011400 con 0x7fba400fe440 2024-03-18T01:18:07.051 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:07.045+0000 7fba27fff700 1 --2- 172.21.15.29:0/3421275186 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7fba280692a0 0x7fba2806b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:18:07.051 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:07.045+0000 7fba3effd700 1 --2- 172.21.15.29:0/3421275186 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7fba280692a0 0x7fba2806b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:18:07.051 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:07.045+0000 7fba27fff700 1 -- 172.21.15.29:0/3421275186 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5464+0+0 (secure 0 0 0) 0x7fba30086f70 con 0x7fba400fe440 2024-03-18T01:18:07.051 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:07.046+0000 7fba3effd700 1 --2- 172.21.15.29:0/3421275186 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7fba280692a0 0x7fba2806b750 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fba3400b970 tx=0x7fba34009e30 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:18:07.051 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:07.050+0000 7fba27fff700 1 -- 172.21.15.29:0/3421275186 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175520 (secure 0 0 0) 0x7fba30056d30 con 0x7fba400fe440 2024-03-18T01:18:07.297 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:07.296+0000 7fba45b57700 1 -- 172.21.15.29:0/3421275186 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fba40113a50 con 0x7fba400fe440 2024-03-18T01:18:07.297 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:07.297+0000 7fba27fff700 1 -- 172.21.15.29:0/3421275186 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 4 v4) v1 ==== 75+0+1247 (secure 0 0 0) 0x7fba300568c0 con 0x7fba400fe440 2024-03-18T01:18:07.297 INFO:teuthology.orchestra.run.smithi029.stdout:e4 2024-03-18T01:18:07.297 INFO:teuthology.orchestra.run.smithi029.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-18T01:18:07.297 INFO:teuthology.orchestra.run.smithi029.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-18T01:18:07.298 INFO:teuthology.orchestra.run.smithi029.stdout:legacy client fscid: 1 2024-03-18T01:18:07.298 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-18T01:18:07.298 INFO:teuthology.orchestra.run.smithi029.stdout:Filesystem 'cephfs' (1) 2024-03-18T01:18:07.298 INFO:teuthology.orchestra.run.smithi029.stdout:fs_name cephfs 2024-03-18T01:18:07.298 INFO:teuthology.orchestra.run.smithi029.stdout:epoch 4 2024-03-18T01:18:07.298 INFO:teuthology.orchestra.run.smithi029.stdout:flags 12 joinable allow_snaps allow_multimds_snaps 2024-03-18T01:18:07.298 INFO:teuthology.orchestra.run.smithi029.stdout:created 2024-03-18T01:18:03.228199+0000 2024-03-18T01:18:07.298 INFO:teuthology.orchestra.run.smithi029.stdout:modified 2024-03-18T01:18:07.242951+0000 2024-03-18T01:18:07.298 INFO:teuthology.orchestra.run.smithi029.stdout:tableserver 0 2024-03-18T01:18:07.298 INFO:teuthology.orchestra.run.smithi029.stdout:root 0 2024-03-18T01:18:07.298 INFO:teuthology.orchestra.run.smithi029.stdout:session_timeout 60 2024-03-18T01:18:07.298 INFO:teuthology.orchestra.run.smithi029.stdout:session_autoclose 300 2024-03-18T01:18:07.298 INFO:teuthology.orchestra.run.smithi029.stdout:max_file_size 1099511627776 2024-03-18T01:18:07.298 INFO:teuthology.orchestra.run.smithi029.stdout:required_client_features {} 2024-03-18T01:18:07.299 INFO:teuthology.orchestra.run.smithi029.stdout:last_failure 0 2024-03-18T01:18:07.299 INFO:teuthology.orchestra.run.smithi029.stdout:last_failure_osd_epoch 0 2024-03-18T01:18:07.299 INFO:teuthology.orchestra.run.smithi029.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-18T01:18:07.299 INFO:teuthology.orchestra.run.smithi029.stdout:max_mds 1 2024-03-18T01:18:07.299 INFO:teuthology.orchestra.run.smithi029.stdout:in 0 2024-03-18T01:18:07.299 INFO:teuthology.orchestra.run.smithi029.stdout:up {0=14470} 2024-03-18T01:18:07.299 INFO:teuthology.orchestra.run.smithi029.stdout:failed 2024-03-18T01:18:07.299 INFO:teuthology.orchestra.run.smithi029.stdout:damaged 2024-03-18T01:18:07.299 INFO:teuthology.orchestra.run.smithi029.stdout:stopped 2024-03-18T01:18:07.299 INFO:teuthology.orchestra.run.smithi029.stdout:data_pools [3] 2024-03-18T01:18:07.299 INFO:teuthology.orchestra.run.smithi029.stdout:metadata_pool 2 2024-03-18T01:18:07.299 INFO:teuthology.orchestra.run.smithi029.stdout:inline_data disabled 2024-03-18T01:18:07.299 INFO:teuthology.orchestra.run.smithi029.stdout:balancer 2024-03-18T01:18:07.299 INFO:teuthology.orchestra.run.smithi029.stdout:standby_count_wanted 0 2024-03-18T01:18:07.299 INFO:teuthology.orchestra.run.smithi029.stdout:[mds.cephfs.smithi029.xuxvlr{0:14470} state up:creating seq 1 addr [v2:172.21.15.29:6826/1784211444,v1:172.21.15.29:6827/1784211444] compat {c=[1],r=[1],i=[7ff]}] 2024-03-18T01:18:07.300 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-18T01:18:07.300 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-18T01:18:07.300 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:07.299+0000 7fba45b57700 1 -- 172.21.15.29:0/3421275186 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7fba280692a0 msgr2=0x7fba2806b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:18:07.300 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:07.299+0000 7fba45b57700 1 --2- 172.21.15.29:0/3421275186 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7fba280692a0 0x7fba2806b750 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fba3400b970 tx=0x7fba34009e30 comp rx=0 tx=0).stop 2024-03-18T01:18:07.301 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:07.299+0000 7fba45b57700 1 -- 172.21.15.29:0/3421275186 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fba400fe440 msgr2=0x7fba4010d900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:18:07.301 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:07.299+0000 7fba45b57700 1 --2- 172.21.15.29:0/3421275186 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fba400fe440 0x7fba4010d900 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fba3000b790 tx=0x7fba30009250 comp rx=0 tx=0).stop 2024-03-18T01:18:07.301 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:07.299+0000 7fba45b57700 1 -- 172.21.15.29:0/3421275186 shutdown_connections 2024-03-18T01:18:07.301 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:07.299+0000 7fba45b57700 1 --2- 172.21.15.29:0/3421275186 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7fba280692a0 0x7fba2806b750 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:18:07.301 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:07.299+0000 7fba45b57700 1 --2- 172.21.15.29:0/3421275186 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fba400fe440 0x7fba4010d900 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:18:07.301 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:07.299+0000 7fba45b57700 1 --2- 172.21.15.29:0/3421275186 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fba400fed90 0x7fba4010de40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:18:07.301 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:07.299+0000 7fba45b57700 1 -- 172.21.15.29:0/3421275186 >> 172.21.15.29:0/3421275186 conn(0x7fba400f6bc0 msgr2=0x7fba400f81d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:18:07.301 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:07.299+0000 7fba45b57700 1 -- 172.21.15.29:0/3421275186 shutdown_connections 2024-03-18T01:18:07.301 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:07.299+0000 7fba45b57700 1 -- 172.21.15.29:0/3421275186 wait complete. 2024-03-18T01:18:07.301 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 4 2024-03-18T01:18:07.953 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-18T01:18:07.963 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi029.front.sepia.ceph.com 2024-03-18T01:18:07.963 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 37f51182-e4c4-11ee-95c9-87774f69a715 -- bash -c 'ceph fs set cephfs max_mds 2' 2024-03-18T01:18:08.226 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/37f51182-e4c4-11ee-95c9-87774f69a715/mon.smithi029/config 2024-03-18T01:18:08.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:18:08 smithi090 ceph-mon[36989]: pgmap v181: 65 pgs: 31 active+clean, 17 creating+peering, 17 unknown; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-18T01:18:08.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:18:08 smithi090 ceph-mon[36989]: mds.? [v2:172.21.15.29:6826/1784211444,v1:172.21.15.29:6827/1784211444] up:boot 2024-03-18T01:18:08.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:18:08 smithi090 ceph-mon[36989]: daemon mds.cephfs.smithi029.xuxvlr assigned to filesystem cephfs as rank 0 (now has 1 ranks) 2024-03-18T01:18:08.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:18:08 smithi090 ceph-mon[36989]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-03-18T01:18:08.527 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:18:08 smithi090 ceph-mon[36989]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-03-18T01:18:08.527 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:18:08 smithi090 ceph-mon[36989]: Cluster is now healthy 2024-03-18T01:18:08.527 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:18:08 smithi090 ceph-mon[36989]: fsmap cephfs:0 1 up:standby 2024-03-18T01:18:08.527 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:18:08 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi029.xuxvlr"}]: dispatch 2024-03-18T01:18:08.527 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:18:08 smithi090 ceph-mon[36989]: fsmap cephfs:1 {0=cephfs.smithi029.xuxvlr=up:creating} 2024-03-18T01:18:08.527 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:18:08 smithi090 ceph-mon[36989]: daemon mds.cephfs.smithi029.xuxvlr is now active in filesystem cephfs as rank 0 2024-03-18T01:18:08.527 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:18:08 smithi090 ceph-mon[36989]: from='client.? 172.21.15.29:0/3421275186' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-18T01:18:08.527 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:18:08 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:18:08.527 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:18:08 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:18:08.527 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:18:08 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:18:08.527 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:18:08 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi090.bohder", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T01:18:08.527 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:18:08 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi090.bohder", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T01:18:08.527 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:18:08 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:18:08.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:18:08 smithi029 ceph-mon[29331]: pgmap v181: 65 pgs: 31 active+clean, 17 creating+peering, 17 unknown; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-18T01:18:08.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:18:08 smithi029 ceph-mon[29331]: mds.? [v2:172.21.15.29:6826/1784211444,v1:172.21.15.29:6827/1784211444] up:boot 2024-03-18T01:18:08.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:18:08 smithi029 ceph-mon[29331]: daemon mds.cephfs.smithi029.xuxvlr assigned to filesystem cephfs as rank 0 (now has 1 ranks) 2024-03-18T01:18:08.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:18:08 smithi029 ceph-mon[29331]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-03-18T01:18:08.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:18:08 smithi029 ceph-mon[29331]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-03-18T01:18:08.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:18:08 smithi029 ceph-mon[29331]: Cluster is now healthy 2024-03-18T01:18:08.570 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:18:08 smithi029 ceph-mon[29331]: fsmap cephfs:0 1 up:standby 2024-03-18T01:18:08.570 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:18:08 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi029.xuxvlr"}]: dispatch 2024-03-18T01:18:08.570 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:18:08 smithi029 ceph-mon[29331]: fsmap cephfs:1 {0=cephfs.smithi029.xuxvlr=up:creating} 2024-03-18T01:18:08.570 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:18:08 smithi029 ceph-mon[29331]: daemon mds.cephfs.smithi029.xuxvlr is now active in filesystem cephfs as rank 0 2024-03-18T01:18:08.570 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:18:08 smithi029 ceph-mon[29331]: from='client.? 172.21.15.29:0/3421275186' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-18T01:18:08.570 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:18:08 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:18:08.570 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:18:08 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:18:08.570 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:18:08 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:18:08.570 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:18:08 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi090.bohder", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T01:18:08.570 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:18:08 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi090.bohder", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T01:18:08.570 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:18:08 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:18:09.473 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:18:09 smithi090 ceph-mon[36989]: Deploying daemon mds.cephfs.smithi090.bohder on smithi090 2024-03-18T01:18:09.474 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:18:09 smithi090 ceph-mon[36989]: mds.? [v2:172.21.15.29:6826/1784211444,v1:172.21.15.29:6827/1784211444] up:active 2024-03-18T01:18:09.474 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:18:09 smithi090 ceph-mon[36989]: fsmap cephfs:1 {0=cephfs.smithi029.xuxvlr=up:active} 2024-03-18T01:18:09.493 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:18:09 smithi029 ceph-mon[29331]: Deploying daemon mds.cephfs.smithi090.bohder on smithi090 2024-03-18T01:18:09.493 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:18:09 smithi029 ceph-mon[29331]: mds.? [v2:172.21.15.29:6826/1784211444,v1:172.21.15.29:6827/1784211444] up:active 2024-03-18T01:18:09.493 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:18:09 smithi029 ceph-mon[29331]: fsmap cephfs:1 {0=cephfs.smithi029.xuxvlr=up:active} 2024-03-18T01:18:10.044 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:10.043+0000 7f4e8bfb2700 1 -- 172.21.15.29:0/2057529190 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4e840fdfe0 msgr2=0x7f4e840fe450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:18:10.044 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:10.043+0000 7f4e8bfb2700 1 --2- 172.21.15.29:0/2057529190 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4e840fdfe0 0x7f4e840fe450 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f4e78004940 tx=0x7f4e78009e30 comp rx=0 tx=0).stop 2024-03-18T01:18:10.044 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:10.043+0000 7f4e8bfb2700 1 -- 172.21.15.29:0/2057529190 shutdown_connections 2024-03-18T01:18:10.045 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:10.043+0000 7f4e8bfb2700 1 --2- 172.21.15.29:0/2057529190 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4e840fdfe0 0x7f4e840fe450 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:18:10.045 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:10.043+0000 7f4e8bfb2700 1 --2- 172.21.15.29:0/2057529190 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4e840fce70 0x7f4e840fd280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:18:10.045 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:10.043+0000 7f4e8bfb2700 1 -- 172.21.15.29:0/2057529190 >> 172.21.15.29:0/2057529190 conn(0x7f4e840f8550 msgr2=0x7f4e840fa960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:18:10.045 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:10.043+0000 7f4e8bfb2700 1 -- 172.21.15.29:0/2057529190 shutdown_connections 2024-03-18T01:18:10.045 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:10.043+0000 7f4e8bfb2700 1 -- 172.21.15.29:0/2057529190 wait complete. 2024-03-18T01:18:10.045 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:10.043+0000 7f4e8bfb2700 1 Processor -- start 2024-03-18T01:18:10.045 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:10.043+0000 7f4e8bfb2700 1 -- start start 2024-03-18T01:18:10.045 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:10.044+0000 7f4e8bfb2700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4e840fce70 0x7f4e84071c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:18:10.045 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:10.044+0000 7f4e8bfb2700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4e840fdfe0 0x7f4e84070310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:18:10.045 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:10.044+0000 7f4e8bfb2700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e84072260 con 0x7f4e840fce70 2024-03-18T01:18:10.045 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:10.044+0000 7f4e8bfb2700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e84070850 con 0x7f4e840fdfe0 2024-03-18T01:18:10.045 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:10.044+0000 7f4e89d4e700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4e840fce70 0x7f4e84071c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:18:10.045 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:10.044+0000 7f4e89d4e700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4e840fce70 0x7f4e84071c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42712/0 (socket says 172.21.15.29:42712) 2024-03-18T01:18:10.045 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:10.044+0000 7f4e89d4e700 1 -- 172.21.15.29:0/2131284984 learned_addr learned my addr 172.21.15.29:0/2131284984 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-18T01:18:10.045 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:10.044+0000 7f4e8954d700 1 --2- 172.21.15.29:0/2131284984 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4e840fdfe0 0x7f4e84070310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:18:10.046 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:10.044+0000 7f4e89d4e700 1 -- 172.21.15.29:0/2131284984 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4e840fdfe0 msgr2=0x7f4e84070310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:18:10.046 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:10.044+0000 7f4e89d4e700 1 --2- 172.21.15.29:0/2131284984 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4e840fdfe0 0x7f4e84070310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:18:10.046 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:10.044+0000 7f4e89d4e700 1 -- 172.21.15.29:0/2131284984 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e78005040 con 0x7f4e840fce70 2024-03-18T01:18:10.047 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:10.044+0000 7f4e89d4e700 1 --2- 172.21.15.29:0/2131284984 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4e840fce70 0x7f4e84071c60 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f4e80002a50 tx=0x7f4e800046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:18:10.047 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:10.045+0000 7f4e767fc700 1 -- 172.21.15.29:0/2131284984 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4e80007cc0 con 0x7f4e840fce70 2024-03-18T01:18:10.047 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:10.045+0000 7f4e8bfb2700 1 -- 172.21.15.29:0/2131284984 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e84070b00 con 0x7f4e840fce70 2024-03-18T01:18:10.047 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:10.045+0000 7f4e767fc700 1 -- 172.21.15.29:0/2131284984 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4e8000f040 con 0x7f4e840fce70 2024-03-18T01:18:10.048 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:10.045+0000 7f4e8bfb2700 1 -- 172.21.15.29:0/2131284984 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e840710c0 con 0x7f4e840fce70 2024-03-18T01:18:10.048 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:10.045+0000 7f4e767fc700 1 -- 172.21.15.29:0/2131284984 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4e8000da30 con 0x7f4e840fce70 2024-03-18T01:18:10.048 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:10.046+0000 7f4e8bfb2700 1 -- 172.21.15.29:0/2131284984 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e680052e0 con 0x7f4e840fce70 2024-03-18T01:18:10.048 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:10.046+0000 7f4e767fc700 1 -- 172.21.15.29:0/2131284984 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f4e800097c0 con 0x7f4e840fce70 2024-03-18T01:18:10.048 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:10.047+0000 7f4e767fc700 1 --2- 172.21.15.29:0/2131284984 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f4e700692a0 0x7f4e7006b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:18:10.048 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:10.047+0000 7f4e767fc700 1 -- 172.21.15.29:0/2131284984 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5464+0+0 (secure 0 0 0) 0x7f4e80086e10 con 0x7f4e840fce70 2024-03-18T01:18:10.048 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:10.047+0000 7f4e8954d700 1 --2- 172.21.15.29:0/2131284984 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f4e700692a0 0x7f4e7006b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:18:10.048 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:10.047+0000 7f4e8954d700 1 --2- 172.21.15.29:0/2131284984 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f4e700692a0 0x7f4e7006b750 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f4e7800b970 tx=0x7f4e78009e30 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:18:10.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:10.052+0000 7f4e767fc700 1 -- 172.21.15.29:0/2131284984 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175520 (secure 0 0 0) 0x7f4e80056bd0 con 0x7f4e840fce70 2024-03-18T01:18:10.301 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:10.301+0000 7f4e8bfb2700 1 -- 172.21.15.29:0/2131284984 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"} v 0) v1 -- 0x7f4e68005b10 con 0x7f4e840fce70 2024-03-18T01:18:10.319 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:18:10 smithi029 ceph-mon[29331]: pgmap v182: 65 pgs: 59 active+clean, 6 creating+peering; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.4 KiB/s wr, 1 op/s 2024-03-18T01:18:10.319 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:18:10 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:18:10.319 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:18:10 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:18:10.319 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:18:10 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:18:10.319 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:18:10 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi029.brxqeh", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T01:18:10.319 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:18:10 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi029.brxqeh", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T01:18:10.319 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:18:10 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:18:10.320 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:18:10 smithi029 ceph-mon[29331]: mds.? [v2:172.21.15.90:6824/1998449484,v1:172.21.15.90:6825/1998449484] up:boot 2024-03-18T01:18:10.320 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:18:10 smithi029 ceph-mon[29331]: fsmap cephfs:1 {0=cephfs.smithi029.xuxvlr=up:active} 1 up:standby 2024-03-18T01:18:10.320 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:18:10 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi090.bohder"}]: dispatch 2024-03-18T01:18:10.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:18:10 smithi090 ceph-mon[36989]: pgmap v182: 65 pgs: 59 active+clean, 6 creating+peering; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.4 KiB/s wr, 1 op/s 2024-03-18T01:18:10.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:18:10 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:18:10.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:18:10 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:18:10.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:18:10 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:18:10.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:18:10 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi029.brxqeh", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T01:18:10.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:18:10 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi029.brxqeh", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T01:18:10.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:18:10 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:18:10.527 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:18:10 smithi090 ceph-mon[36989]: mds.? [v2:172.21.15.90:6824/1998449484,v1:172.21.15.90:6825/1998449484] up:boot 2024-03-18T01:18:10.527 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:18:10 smithi090 ceph-mon[36989]: fsmap cephfs:1 {0=cephfs.smithi029.xuxvlr=up:active} 1 up:standby 2024-03-18T01:18:10.527 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:18:10 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi090.bohder"}]: dispatch 2024-03-18T01:18:11.261 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:11.260+0000 7f4e767fc700 1 -- 172.21.15.29:0/2131284984 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"}]=0 v7) v1 ==== 105+0+0 (secure 0 0 0) 0x7f4e80056760 con 0x7f4e840fce70 2024-03-18T01:18:11.263 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:11.263+0000 7f4e8bfb2700 1 -- 172.21.15.29:0/2131284984 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f4e700692a0 msgr2=0x7f4e7006b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:18:11.263 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:11.263+0000 7f4e8bfb2700 1 --2- 172.21.15.29:0/2131284984 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f4e700692a0 0x7f4e7006b750 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f4e7800b970 tx=0x7f4e78009e30 comp rx=0 tx=0).stop 2024-03-18T01:18:11.263 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:11.263+0000 7f4e8bfb2700 1 -- 172.21.15.29:0/2131284984 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4e840fce70 msgr2=0x7f4e84071c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:18:11.263 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:11.263+0000 7f4e8bfb2700 1 --2- 172.21.15.29:0/2131284984 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4e840fce70 0x7f4e84071c60 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f4e80002a50 tx=0x7f4e800046b0 comp rx=0 tx=0).stop 2024-03-18T01:18:11.263 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:11.263+0000 7f4e8bfb2700 1 -- 172.21.15.29:0/2131284984 shutdown_connections 2024-03-18T01:18:11.263 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:11.263+0000 7f4e8bfb2700 1 --2- 172.21.15.29:0/2131284984 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f4e700692a0 0x7f4e7006b750 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:18:11.264 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:11.263+0000 7f4e8bfb2700 1 --2- 172.21.15.29:0/2131284984 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4e840fce70 0x7f4e84071c60 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:18:11.264 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:11.263+0000 7f4e8bfb2700 1 --2- 172.21.15.29:0/2131284984 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4e840fdfe0 0x7f4e84070310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:18:11.264 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:11.263+0000 7f4e8bfb2700 1 -- 172.21.15.29:0/2131284984 >> 172.21.15.29:0/2131284984 conn(0x7f4e840f8550 msgr2=0x7f4e840f9c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:18:11.264 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:11.263+0000 7f4e8bfb2700 1 -- 172.21.15.29:0/2131284984 shutdown_connections 2024-03-18T01:18:11.264 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:11.263+0000 7f4e8bfb2700 1 -- 172.21.15.29:0/2131284984 wait complete. 2024-03-18T01:18:11.449 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:18:11 smithi029 ceph-mon[29331]: Deploying daemon mds.cephfs.smithi029.brxqeh on smithi029 2024-03-18T01:18:11.449 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:18:11 smithi029 ceph-mon[29331]: from='client.? 172.21.15.29:0/2131284984' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"}]: dispatch 2024-03-18T01:18:11.450 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:18:11 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-18T01:18:11.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:18:11 smithi090 ceph-mon[36989]: Deploying daemon mds.cephfs.smithi029.brxqeh on smithi029 2024-03-18T01:18:11.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:18:11 smithi090 ceph-mon[36989]: from='client.? 172.21.15.29:0/2131284984' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"}]: dispatch 2024-03-18T01:18:11.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:18:11 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-18T01:18:12.300 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-18T01:18:12.310 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi029.front.sepia.ceph.com 2024-03-18T01:18:12.310 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 37f51182-e4c4-11ee-95c9-87774f69a715 -- bash -c 'ceph fs set cephfs allow_standby_replay true' 2024-03-18T01:18:12.519 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:18:12 smithi029 ceph-mon[29331]: pgmap v183: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.4 KiB/s wr, 5 op/s 2024-03-18T01:18:12.520 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:18:12 smithi029 ceph-mon[29331]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-03-18T01:18:12.520 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:18:12 smithi029 ceph-mon[29331]: from='client.? 172.21.15.29:0/2131284984' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"}]': finished 2024-03-18T01:18:12.520 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:18:12 smithi029 ceph-mon[29331]: daemon mds.cephfs.smithi090.bohder assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-03-18T01:18:12.520 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:18:12 smithi029 ceph-mon[29331]: Health check failed: insufficient standby MDS daemons available (MDS_INSUFFICIENT_STANDBY) 2024-03-18T01:18:12.520 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:18:12 smithi029 ceph-mon[29331]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-03-18T01:18:12.520 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:18:12 smithi029 ceph-mon[29331]: fsmap cephfs:1 {0=cephfs.smithi029.xuxvlr=up:active} 1 up:standby 2024-03-18T01:18:12.520 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:18:12 smithi029 ceph-mon[29331]: fsmap cephfs:2 {0=cephfs.smithi029.xuxvlr=up:active,1=cephfs.smithi090.bohder=up:creating} 2024-03-18T01:18:12.520 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:18:12 smithi029 ceph-mon[29331]: daemon mds.cephfs.smithi090.bohder is now active in filesystem cephfs as rank 1 2024-03-18T01:18:12.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:18:12 smithi090 ceph-mon[36989]: pgmap v183: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.4 KiB/s wr, 5 op/s 2024-03-18T01:18:12.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:18:12 smithi090 ceph-mon[36989]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-03-18T01:18:12.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:18:12 smithi090 ceph-mon[36989]: from='client.? 172.21.15.29:0/2131284984' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"}]': finished 2024-03-18T01:18:12.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:18:12 smithi090 ceph-mon[36989]: daemon mds.cephfs.smithi090.bohder assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-03-18T01:18:12.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:18:12 smithi090 ceph-mon[36989]: Health check failed: insufficient standby MDS daemons available (MDS_INSUFFICIENT_STANDBY) 2024-03-18T01:18:12.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:18:12 smithi090 ceph-mon[36989]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-03-18T01:18:12.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:18:12 smithi090 ceph-mon[36989]: fsmap cephfs:1 {0=cephfs.smithi029.xuxvlr=up:active} 1 up:standby 2024-03-18T01:18:12.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:18:12 smithi090 ceph-mon[36989]: fsmap cephfs:2 {0=cephfs.smithi029.xuxvlr=up:active,1=cephfs.smithi090.bohder=up:creating} 2024-03-18T01:18:12.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:18:12 smithi090 ceph-mon[36989]: daemon mds.cephfs.smithi090.bohder is now active in filesystem cephfs as rank 1 2024-03-18T01:18:12.945 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/37f51182-e4c4-11ee-95c9-87774f69a715/mon.smithi029/config 2024-03-18T01:18:13.319 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:18:13 smithi029 ceph-mon[29331]: mds.? [v2:172.21.15.90:6824/1998449484,v1:172.21.15.90:6825/1998449484] up:active 2024-03-18T01:18:13.319 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:18:13 smithi029 ceph-mon[29331]: fsmap cephfs:2 {0=cephfs.smithi029.xuxvlr=up:active,1=cephfs.smithi090.bohder=up:active} 2024-03-18T01:18:13.319 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:18:13 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:18:13.319 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:18:13 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:18:13.319 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:18:13 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:18:13.319 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:18:13 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi090.trsnxz", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T01:18:13.319 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:18:13 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi090.trsnxz", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T01:18:13.320 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:18:13 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:18:13.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:18:13 smithi090 ceph-mon[36989]: mds.? [v2:172.21.15.90:6824/1998449484,v1:172.21.15.90:6825/1998449484] up:active 2024-03-18T01:18:13.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:18:13 smithi090 ceph-mon[36989]: fsmap cephfs:2 {0=cephfs.smithi029.xuxvlr=up:active,1=cephfs.smithi090.bohder=up:active} 2024-03-18T01:18:13.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:18:13 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:18:13.527 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:18:13 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:18:13.527 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:18:13 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:18:13.527 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:18:13 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi090.trsnxz", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T01:18:13.527 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:18:13 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi090.trsnxz", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T01:18:13.527 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:18:13 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:18:14.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:18:14 smithi090 ceph-mon[36989]: Deploying daemon mds.cephfs.smithi090.trsnxz on smithi090 2024-03-18T01:18:14.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:18:14 smithi090 ceph-mon[36989]: pgmap v184: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.6 KiB/s wr, 6 op/s 2024-03-18T01:18:14.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:18:14 smithi090 ceph-mon[36989]: Health check cleared: MDS_INSUFFICIENT_STANDBY (was: insufficient standby MDS daemons available) 2024-03-18T01:18:14.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:18:14 smithi090 ceph-mon[36989]: Cluster is now healthy 2024-03-18T01:18:14.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:18:14 smithi090 ceph-mon[36989]: mds.? [v2:172.21.15.29:6828/1299751620,v1:172.21.15.29:6829/1299751620] up:boot 2024-03-18T01:18:14.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:18:14 smithi090 ceph-mon[36989]: fsmap cephfs:2 {0=cephfs.smithi029.xuxvlr=up:active,1=cephfs.smithi090.bohder=up:active} 1 up:standby 2024-03-18T01:18:14.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:18:14 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi029.brxqeh"}]: dispatch 2024-03-18T01:18:14.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:18:14 smithi029 ceph-mon[29331]: Deploying daemon mds.cephfs.smithi090.trsnxz on smithi090 2024-03-18T01:18:14.570 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:18:14 smithi029 ceph-mon[29331]: pgmap v184: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.6 KiB/s wr, 6 op/s 2024-03-18T01:18:14.570 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:18:14 smithi029 ceph-mon[29331]: Health check cleared: MDS_INSUFFICIENT_STANDBY (was: insufficient standby MDS daemons available) 2024-03-18T01:18:14.570 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:18:14 smithi029 ceph-mon[29331]: Cluster is now healthy 2024-03-18T01:18:14.570 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:18:14 smithi029 ceph-mon[29331]: mds.? [v2:172.21.15.29:6828/1299751620,v1:172.21.15.29:6829/1299751620] up:boot 2024-03-18T01:18:14.570 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:18:14 smithi029 ceph-mon[29331]: fsmap cephfs:2 {0=cephfs.smithi029.xuxvlr=up:active,1=cephfs.smithi090.bohder=up:active} 1 up:standby 2024-03-18T01:18:14.570 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:18:14 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi029.brxqeh"}]: dispatch 2024-03-18T01:18:14.596 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:14.593+0000 7fd93ec30700 1 -- 172.21.15.29:0/939467739 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd9380fce70 msgr2=0x7fd9380fd280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:18:14.596 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:14.593+0000 7fd93ec30700 1 --2- 172.21.15.29:0/939467739 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd9380fce70 0x7fd9380fd280 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fd92c004940 tx=0x7fd92c009e30 comp rx=0 tx=0).stop 2024-03-18T01:18:14.596 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:14.594+0000 7fd93ec30700 1 -- 172.21.15.29:0/939467739 shutdown_connections 2024-03-18T01:18:14.596 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:14.594+0000 7fd93ec30700 1 --2- 172.21.15.29:0/939467739 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd9380fdfe0 0x7fd9380fe450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:18:14.596 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:14.594+0000 7fd93ec30700 1 --2- 172.21.15.29:0/939467739 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd9380fce70 0x7fd9380fd280 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:18:14.596 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:14.594+0000 7fd93ec30700 1 -- 172.21.15.29:0/939467739 >> 172.21.15.29:0/939467739 conn(0x7fd9380f8550 msgr2=0x7fd9380fa960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:18:14.596 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:14.594+0000 7fd93ec30700 1 -- 172.21.15.29:0/939467739 shutdown_connections 2024-03-18T01:18:14.596 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:14.594+0000 7fd93ec30700 1 -- 172.21.15.29:0/939467739 wait complete. 2024-03-18T01:18:14.596 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:14.594+0000 7fd93ec30700 1 Processor -- start 2024-03-18T01:18:14.597 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:14.594+0000 7fd93ec30700 1 -- start start 2024-03-18T01:18:14.597 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:14.595+0000 7fd93ec30700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd9380fce70 0x7fd938071c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:18:14.597 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:14.595+0000 7fd93ec30700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd9380fdfe0 0x7fd938070310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:18:14.597 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:14.595+0000 7fd93ec30700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd938072260 con 0x7fd9380fdfe0 2024-03-18T01:18:14.597 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:14.595+0000 7fd93ec30700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd938070850 con 0x7fd9380fce70 2024-03-18T01:18:14.597 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:14.595+0000 7fd937fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd9380fdfe0 0x7fd938070310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:18:14.597 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:14.595+0000 7fd937fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd9380fdfe0 0x7fd938070310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:58964/0 (socket says 172.21.15.29:58964) 2024-03-18T01:18:14.597 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:14.595+0000 7fd937fff700 1 -- 172.21.15.29:0/756168547 learned_addr learned my addr 172.21.15.29:0/756168547 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-18T01:18:14.597 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:14.595+0000 7fd93c9cc700 1 --2- 172.21.15.29:0/756168547 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd9380fce70 0x7fd938071c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:18:14.597 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:14.595+0000 7fd937fff700 1 -- 172.21.15.29:0/756168547 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd9380fce70 msgr2=0x7fd938071c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:18:14.597 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:14.595+0000 7fd937fff700 1 --2- 172.21.15.29:0/756168547 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd9380fce70 0x7fd938071c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:18:14.597 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:14.595+0000 7fd937fff700 1 -- 172.21.15.29:0/756168547 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd92c005040 con 0x7fd9380fdfe0 2024-03-18T01:18:14.598 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:14.595+0000 7fd937fff700 1 --2- 172.21.15.29:0/756168547 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd9380fdfe0 0x7fd938070310 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fd92400eb40 tx=0x7fd92400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:18:14.598 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:14.595+0000 7fd9357fa700 1 -- 172.21.15.29:0/756168547 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd924007cc0 con 0x7fd9380fdfe0 2024-03-18T01:18:14.598 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:14.595+0000 7fd93ec30700 1 -- 172.21.15.29:0/756168547 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd938070aa0 con 0x7fd9380fdfe0 2024-03-18T01:18:14.598 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:14.595+0000 7fd9357fa700 1 -- 172.21.15.29:0/756168547 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd92400f040 con 0x7fd9380fdfe0 2024-03-18T01:18:14.598 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:14.595+0000 7fd9357fa700 1 -- 172.21.15.29:0/756168547 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd92400ca10 con 0x7fd9380fdfe0 2024-03-18T01:18:14.598 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:14.596+0000 7fd93ec30700 1 -- 172.21.15.29:0/756168547 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd938070fe0 con 0x7fd9380fdfe0 2024-03-18T01:18:14.599 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:14.596+0000 7fd93ec30700 1 -- 172.21.15.29:0/756168547 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd93804c710 con 0x7fd9380fdfe0 2024-03-18T01:18:14.599 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:14.597+0000 7fd9357fa700 1 -- 172.21.15.29:0/756168547 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7fd92400cc80 con 0x7fd9380fdfe0 2024-03-18T01:18:14.599 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:14.597+0000 7fd9357fa700 1 --2- 172.21.15.29:0/756168547 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7fd9280692a0 0x7fd92806b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:18:14.599 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:14.598+0000 7fd93c9cc700 1 --2- 172.21.15.29:0/756168547 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7fd9280692a0 0x7fd92806b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:18:14.599 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:14.598+0000 7fd9357fa700 1 -- 172.21.15.29:0/756168547 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5464+0+0 (secure 0 0 0) 0x7fd924086d80 con 0x7fd9380fdfe0 2024-03-18T01:18:14.599 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:14.598+0000 7fd93c9cc700 1 --2- 172.21.15.29:0/756168547 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7fd9280692a0 0x7fd92806b750 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fd92c009b30 tx=0x7fd92c009e30 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:18:14.602 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:14.602+0000 7fd9357fa700 1 -- 172.21.15.29:0/756168547 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175520 (secure 0 0 0) 0x7fd92404ddb0 con 0x7fd9380fdfe0 2024-03-18T01:18:14.839 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:14.839+0000 7fd93ec30700 1 -- 172.21.15.29:0/756168547 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"} v 0) v1 -- 0x7fd9381058d0 con 0x7fd9380fdfe0 2024-03-18T01:18:15.111 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:15.110+0000 7fd9357fa700 1 -- 172.21.15.29:0/756168547 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]=0 v11) v1 ==== 121+0+0 (secure 0 0 0) 0x7fd924056b70 con 0x7fd9380fdfe0 2024-03-18T01:18:15.114 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:15.113+0000 7fd93ec30700 1 -- 172.21.15.29:0/756168547 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7fd9280692a0 msgr2=0x7fd92806b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:18:15.114 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:15.113+0000 7fd93ec30700 1 --2- 172.21.15.29:0/756168547 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7fd9280692a0 0x7fd92806b750 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fd92c009b30 tx=0x7fd92c009e30 comp rx=0 tx=0).stop 2024-03-18T01:18:15.114 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:15.113+0000 7fd93ec30700 1 -- 172.21.15.29:0/756168547 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd9380fdfe0 msgr2=0x7fd938070310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:18:15.114 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:15.113+0000 7fd93ec30700 1 --2- 172.21.15.29:0/756168547 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd9380fdfe0 0x7fd938070310 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fd92400eb40 tx=0x7fd92400b040 comp rx=0 tx=0).stop 2024-03-18T01:18:15.114 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:15.113+0000 7fd93ec30700 1 -- 172.21.15.29:0/756168547 shutdown_connections 2024-03-18T01:18:15.115 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:15.113+0000 7fd93ec30700 1 --2- 172.21.15.29:0/756168547 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7fd9280692a0 0x7fd92806b750 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:18:15.115 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:15.114+0000 7fd93ec30700 1 --2- 172.21.15.29:0/756168547 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd9380fce70 0x7fd938071c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:18:15.115 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:15.114+0000 7fd93ec30700 1 --2- 172.21.15.29:0/756168547 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd9380fdfe0 0x7fd938070310 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:18:15.115 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:15.114+0000 7fd93ec30700 1 -- 172.21.15.29:0/756168547 >> 172.21.15.29:0/756168547 conn(0x7fd9380f8550 msgr2=0x7fd9380f9bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:18:15.115 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:15.114+0000 7fd93ec30700 1 -- 172.21.15.29:0/756168547 shutdown_connections 2024-03-18T01:18:15.115 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:15.114+0000 7fd93ec30700 1 -- 172.21.15.29:0/756168547 wait complete. 2024-03-18T01:18:15.393 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:18:15 smithi090 ceph-mon[36989]: from='client.? 172.21.15.29:0/756168547' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-03-18T01:18:15.394 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:18:15 smithi090 ceph-mon[36989]: from='client.? 172.21.15.29:0/756168547' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-03-18T01:18:15.394 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:18:15 smithi090 ceph-mon[36989]: fsmap cephfs:2 {0=cephfs.smithi029.xuxvlr=up:active,1=cephfs.smithi090.bohder=up:active} 1 up:standby-replay 2024-03-18T01:18:15.472 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:18:15 smithi029 ceph-mon[29331]: from='client.? 172.21.15.29:0/756168547' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-03-18T01:18:15.472 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:18:15 smithi029 ceph-mon[29331]: from='client.? 172.21.15.29:0/756168547' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-03-18T01:18:15.473 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:18:15 smithi029 ceph-mon[29331]: fsmap cephfs:2 {0=cephfs.smithi029.xuxvlr=up:active,1=cephfs.smithi090.bohder=up:active} 1 up:standby-replay 2024-03-18T01:18:15.852 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-18T01:18:15.863 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi029.front.sepia.ceph.com 2024-03-18T01:18:15.864 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 37f51182-e4c4-11ee-95c9-87774f69a715 -- bash -c 'ceph fs set cephfs inline_data true --yes-i-really-really-mean-it' 2024-03-18T01:18:16.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:18:16 smithi090 ceph-mon[36989]: pgmap v185: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.4 KiB/s wr, 7 op/s 2024-03-18T01:18:16.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:18:16 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:18:16.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:18:16 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:18:16.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:18:16 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:18:16.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:18:16 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:18:16.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:18:16 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:18:16.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:18:16 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T01:18:16.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:18:16 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:18:16.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:18:16 smithi029 ceph-mon[29331]: pgmap v185: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.4 KiB/s wr, 7 op/s 2024-03-18T01:18:16.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:18:16 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:18:16.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:18:16 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:18:16.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:18:16 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:18:16.570 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:18:16 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:18:16.570 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:18:16 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:18:16.570 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:18:16 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T01:18:16.570 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:18:16 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:18:16.674 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/37f51182-e4c4-11ee-95c9-87774f69a715/mon.smithi029/config 2024-03-18T01:18:17.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:18:17 smithi090 ceph-mon[36989]: mds.? [v2:172.21.15.90:6826/439410941,v1:172.21.15.90:6827/439410941] up:boot 2024-03-18T01:18:17.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:18:17 smithi090 ceph-mon[36989]: fsmap cephfs:2 {0=cephfs.smithi029.xuxvlr=up:active,1=cephfs.smithi090.bohder=up:active} 1 up:standby-replay 1 up:standby 2024-03-18T01:18:17.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:18:17 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi090.trsnxz"}]: dispatch 2024-03-18T01:18:17.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:18:17 smithi090 ceph-mon[36989]: fsmap cephfs:2 {0=cephfs.smithi029.xuxvlr=up:active,1=cephfs.smithi090.bohder=up:active} 2 up:standby-replay 2024-03-18T01:18:17.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:18:17 smithi029 ceph-mon[29331]: mds.? [v2:172.21.15.90:6826/439410941,v1:172.21.15.90:6827/439410941] up:boot 2024-03-18T01:18:17.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:18:17 smithi029 ceph-mon[29331]: fsmap cephfs:2 {0=cephfs.smithi029.xuxvlr=up:active,1=cephfs.smithi090.bohder=up:active} 1 up:standby-replay 1 up:standby 2024-03-18T01:18:17.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:18:17 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi090.trsnxz"}]: dispatch 2024-03-18T01:18:17.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:18:17 smithi029 ceph-mon[29331]: fsmap cephfs:2 {0=cephfs.smithi029.xuxvlr=up:active,1=cephfs.smithi090.bohder=up:active} 2 up:standby-replay 2024-03-18T01:18:18.319 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:18:18 smithi029 ceph-mon[29331]: pgmap v186: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 348 B/s rd, 2.0 KiB/s wr, 7 op/s 2024-03-18T01:18:18.319 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:18:18 smithi029 ceph-mon[29331]: mds.? [v2:172.21.15.29:6828/1299751620,v1:172.21.15.29:6829/1299751620] up:standby-replay 2024-03-18T01:18:18.319 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:18:18 smithi029 ceph-mon[29331]: Dropping low affinity active daemon mds.cephfs.smithi029.xuxvlr in favor of higher affinity standby. 2024-03-18T01:18:18.319 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:18:18 smithi029 ceph-mon[29331]: Replacing daemon mds.cephfs.smithi029.xuxvlr as rank 0 with standby daemon mds.cephfs.smithi029.brxqeh 2024-03-18T01:18:18.319 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:18:18 smithi029 ceph-mon[29331]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-18T01:18:18.319 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:18:18 smithi029 ceph-mon[29331]: fsmap cephfs:2 {0=cephfs.smithi029.xuxvlr=up:active,1=cephfs.smithi090.bohder=up:active} 2 up:standby-replay 2024-03-18T01:18:18.319 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:18:18 smithi029 ceph-mon[29331]: osdmap e46: 6 total, 6 up, 6 in 2024-03-18T01:18:18.319 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:18:18 smithi029 ceph-mon[29331]: fsmap cephfs:2/2 {0=cephfs.smithi029.brxqeh=up:replay,1=cephfs.smithi090.bohder=up:active} 1 up:standby-replay 2024-03-18T01:18:18.319 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:18:18 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:18:18.320 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:18:18 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:18:18.776 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:18:18 smithi090 ceph-mon[36989]: pgmap v186: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 348 B/s rd, 2.0 KiB/s wr, 7 op/s 2024-03-18T01:18:18.776 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:18:18 smithi090 ceph-mon[36989]: mds.? [v2:172.21.15.29:6828/1299751620,v1:172.21.15.29:6829/1299751620] up:standby-replay 2024-03-18T01:18:18.776 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:18:18 smithi090 ceph-mon[36989]: Dropping low affinity active daemon mds.cephfs.smithi029.xuxvlr in favor of higher affinity standby. 2024-03-18T01:18:18.776 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:18:18 smithi090 ceph-mon[36989]: Replacing daemon mds.cephfs.smithi029.xuxvlr as rank 0 with standby daemon mds.cephfs.smithi029.brxqeh 2024-03-18T01:18:18.777 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:18:18 smithi090 ceph-mon[36989]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-18T01:18:18.777 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:18:18 smithi090 ceph-mon[36989]: fsmap cephfs:2 {0=cephfs.smithi029.xuxvlr=up:active,1=cephfs.smithi090.bohder=up:active} 2 up:standby-replay 2024-03-18T01:18:18.777 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:18:18 smithi090 ceph-mon[36989]: osdmap e46: 6 total, 6 up, 6 in 2024-03-18T01:18:18.777 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:18:18 smithi090 ceph-mon[36989]: fsmap cephfs:2/2 {0=cephfs.smithi029.brxqeh=up:replay,1=cephfs.smithi090.bohder=up:active} 1 up:standby-replay 2024-03-18T01:18:18.777 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:18:18 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:18:18.777 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:18:18 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:18:18.939 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:18.933+0000 7f0d7bc20700 1 -- 172.21.15.29:0/129158179 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0d740fdfe0 msgr2=0x7f0d740fe450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:18:18.940 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:18.933+0000 7f0d7bc20700 1 --2- 172.21.15.29:0/129158179 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0d740fdfe0 0x7f0d740fe450 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f0d64004940 tx=0x7f0d64009e30 comp rx=0 tx=0).stop 2024-03-18T01:18:18.940 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:18.934+0000 7f0d7bc20700 1 -- 172.21.15.29:0/129158179 shutdown_connections 2024-03-18T01:18:18.940 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:18.934+0000 7f0d7bc20700 1 --2- 172.21.15.29:0/129158179 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0d740fdfe0 0x7f0d740fe450 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:18:18.940 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:18.934+0000 7f0d7bc20700 1 --2- 172.21.15.29:0/129158179 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0d740fce70 0x7f0d740fd280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:18:18.940 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:18.934+0000 7f0d7bc20700 1 -- 172.21.15.29:0/129158179 >> 172.21.15.29:0/129158179 conn(0x7f0d740f8550 msgr2=0x7f0d740fa960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:18:18.940 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:18.934+0000 7f0d7bc20700 1 -- 172.21.15.29:0/129158179 shutdown_connections 2024-03-18T01:18:18.940 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:18.934+0000 7f0d7bc20700 1 -- 172.21.15.29:0/129158179 wait complete. 2024-03-18T01:18:18.940 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:18.934+0000 7f0d7bc20700 1 Processor -- start 2024-03-18T01:18:18.940 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:18.934+0000 7f0d7bc20700 1 -- start start 2024-03-18T01:18:18.940 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:18.934+0000 7f0d7bc20700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0d740fce70 0x7f0d7410d940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:18:18.941 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:18.935+0000 7f0d7bc20700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0d740fdfe0 0x7f0d7410de80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:18:18.941 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:18.935+0000 7f0d7bc20700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d7410e510 con 0x7f0d740fdfe0 2024-03-18T01:18:18.941 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:18.935+0000 7f0d7bc20700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d74112eb0 con 0x7f0d740fce70 2024-03-18T01:18:18.941 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:18.935+0000 7f0d791bb700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0d740fdfe0 0x7f0d7410de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:18:18.941 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:18.935+0000 7f0d791bb700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0d740fdfe0 0x7f0d7410de80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:58990/0 (socket says 172.21.15.29:58990) 2024-03-18T01:18:18.941 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:18.935+0000 7f0d791bb700 1 -- 172.21.15.29:0/2117714677 learned_addr learned my addr 172.21.15.29:0/2117714677 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-18T01:18:18.941 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:18.935+0000 7f0d799bc700 1 --2- 172.21.15.29:0/2117714677 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0d740fce70 0x7f0d7410d940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:18:18.941 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:18.935+0000 7f0d791bb700 1 -- 172.21.15.29:0/2117714677 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0d740fce70 msgr2=0x7f0d7410d940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:18:18.941 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:18.935+0000 7f0d791bb700 1 --2- 172.21.15.29:0/2117714677 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0d740fce70 0x7f0d7410d940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:18:18.941 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:18.935+0000 7f0d791bb700 1 -- 172.21.15.29:0/2117714677 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d64005040 con 0x7f0d740fdfe0 2024-03-18T01:18:18.941 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:18.935+0000 7f0d791bb700 1 --2- 172.21.15.29:0/2117714677 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0d740fdfe0 0x7f0d7410de80 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f0d64009150 tx=0x7f0d64009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:18:18.941 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:18.935+0000 7f0d6a7fc700 1 -- 172.21.15.29:0/2117714677 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0d6400e040 con 0x7f0d740fdfe0 2024-03-18T01:18:18.942 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:18.935+0000 7f0d6a7fc700 1 -- 172.21.15.29:0/2117714677 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0d6400bbf0 con 0x7f0d740fdfe0 2024-03-18T01:18:18.942 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:18.935+0000 7f0d7bc20700 1 -- 172.21.15.29:0/2117714677 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d741130b0 con 0x7f0d740fdfe0 2024-03-18T01:18:18.942 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:18.935+0000 7f0d6a7fc700 1 -- 172.21.15.29:0/2117714677 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0d6401c630 con 0x7f0d740fdfe0 2024-03-18T01:18:18.942 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:18.935+0000 7f0d7bc20700 1 -- 172.21.15.29:0/2117714677 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d74113620 con 0x7f0d740fdfe0 2024-03-18T01:18:18.942 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:18.936+0000 7f0d7bc20700 1 -- 172.21.15.29:0/2117714677 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d7404c710 con 0x7f0d740fdfe0 2024-03-18T01:18:18.942 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:18.937+0000 7f0d6a7fc700 1 -- 172.21.15.29:0/2117714677 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f0d6400bd60 con 0x7f0d740fdfe0 2024-03-18T01:18:18.942 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:18.937+0000 7f0d6a7fc700 1 --2- 172.21.15.29:0/2117714677 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f0d600692a0 0x7f0d6006b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:18:18.942 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:18.937+0000 7f0d799bc700 1 --2- 172.21.15.29:0/2117714677 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f0d600692a0 0x7f0d6006b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:18:18.943 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:18.938+0000 7f0d6a7fc700 1 -- 172.21.15.29:0/2117714677 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f0d64088540 con 0x7f0d740fdfe0 2024-03-18T01:18:18.943 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:18.938+0000 7f0d799bc700 1 --2- 172.21.15.29:0/2117714677 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f0d600692a0 0x7f0d6006b750 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f0d70006fd0 tx=0x7f0d70008040 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:18:18.944 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:18.942+0000 7f0d6a7fc700 1 -- 172.21.15.29:0/2117714677 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175520 (secure 0 0 0) 0x7f0d64058400 con 0x7f0d740fdfe0 2024-03-18T01:18:19.193 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:19.192+0000 7f0d7bc20700 1 -- 172.21.15.29:0/2117714677 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "true", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f0d74113ad0 con 0x7f0d740fdfe0 2024-03-18T01:18:19.291 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:19.290+0000 7f0d6a7fc700 1 -- 172.21.15.29:0/2117714677 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "true", "yes_i_really_really_mean_it": true}]=0 inline data enabled v17) v1 ==== 168+0+0 (secure 0 0 0) 0x7f0d64057f90 con 0x7f0d740fdfe0 2024-03-18T01:18:19.293 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:19.292+0000 7f0d7bc20700 1 -- 172.21.15.29:0/2117714677 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f0d600692a0 msgr2=0x7f0d6006b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:18:19.293 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:19.292+0000 7f0d7bc20700 1 --2- 172.21.15.29:0/2117714677 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f0d600692a0 0x7f0d6006b750 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f0d70006fd0 tx=0x7f0d70008040 comp rx=0 tx=0).stop 2024-03-18T01:18:19.293 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:19.292+0000 7f0d7bc20700 1 -- 172.21.15.29:0/2117714677 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0d740fdfe0 msgr2=0x7f0d7410de80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:18:19.293 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:19.292+0000 7f0d7bc20700 1 --2- 172.21.15.29:0/2117714677 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0d740fdfe0 0x7f0d7410de80 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f0d64009150 tx=0x7f0d64009e30 comp rx=0 tx=0).stop 2024-03-18T01:18:19.293 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:19.292+0000 7f0d7bc20700 1 -- 172.21.15.29:0/2117714677 shutdown_connections 2024-03-18T01:18:19.293 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:19.293+0000 7f0d7bc20700 1 --2- 172.21.15.29:0/2117714677 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f0d600692a0 0x7f0d6006b750 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:18:19.293 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:19.293+0000 7f0d7bc20700 1 --2- 172.21.15.29:0/2117714677 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0d740fce70 0x7f0d7410d940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:18:19.293 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:19.293+0000 7f0d7bc20700 1 --2- 172.21.15.29:0/2117714677 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0d740fdfe0 0x7f0d7410de80 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:18:19.293 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:19.293+0000 7f0d7bc20700 1 -- 172.21.15.29:0/2117714677 >> 172.21.15.29:0/2117714677 conn(0x7f0d740f8550 msgr2=0x7f0d740f9b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:18:19.293 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:19.293+0000 7f0d7bc20700 1 -- 172.21.15.29:0/2117714677 shutdown_connections 2024-03-18T01:18:19.294 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:19.293+0000 7f0d7bc20700 1 -- 172.21.15.29:0/2117714677 wait complete. 2024-03-18T01:18:19.294 INFO:teuthology.orchestra.run.smithi029.stderr:inline data enabled 2024-03-18T01:18:19.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:18:19 smithi029 ceph-mon[29331]: mds.? [v2:172.21.15.29:6826/890079928,v1:172.21.15.29:6827/890079928] up:boot 2024-03-18T01:18:19.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:18:19 smithi029 ceph-mon[29331]: mds.? [v2:172.21.15.29:6828/1299751620,v1:172.21.15.29:6829/1299751620] up:resolve 2024-03-18T01:18:19.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:18:19 smithi029 ceph-mon[29331]: fsmap cephfs:2/2 {0=cephfs.smithi029.brxqeh=up:resolve,1=cephfs.smithi090.bohder=up:active} 1 up:standby-replay 1 up:standby 2024-03-18T01:18:19.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:18:19 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi029.xuxvlr"}]: dispatch 2024-03-18T01:18:19.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:18:19 smithi029 ceph-mon[29331]: from='client.? 172.21.15.29:0/2117714677' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "true", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-18T01:18:19.776 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:18:19 smithi090 ceph-mon[36989]: mds.? [v2:172.21.15.29:6826/890079928,v1:172.21.15.29:6827/890079928] up:boot 2024-03-18T01:18:19.776 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:18:19 smithi090 ceph-mon[36989]: mds.? [v2:172.21.15.29:6828/1299751620,v1:172.21.15.29:6829/1299751620] up:resolve 2024-03-18T01:18:19.776 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:18:19 smithi090 ceph-mon[36989]: fsmap cephfs:2/2 {0=cephfs.smithi029.brxqeh=up:resolve,1=cephfs.smithi090.bohder=up:active} 1 up:standby-replay 1 up:standby 2024-03-18T01:18:19.776 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:18:19 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi029.xuxvlr"}]: dispatch 2024-03-18T01:18:19.776 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:18:19 smithi090 ceph-mon[36989]: from='client.? 172.21.15.29:0/2117714677' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "true", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-18T01:18:19.911 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-18T01:18:19.921 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi029.front.sepia.ceph.com 2024-03-18T01:18:19.922 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 37f51182-e4c4-11ee-95c9-87774f69a715 -- bash -c 'ceph fs dump' 2024-03-18T01:18:20.306 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/37f51182-e4c4-11ee-95c9-87774f69a715/mon.smithi029/config 2024-03-18T01:18:20.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:18:20 smithi029 ceph-mon[29331]: pgmap v188: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.1 KiB/s rd, 1.6 KiB/s wr, 8 op/s 2024-03-18T01:18:20.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:18:20 smithi029 ceph-mon[29331]: Health check failed: 1 filesystem with deprecated feature inline_data (FS_INLINE_DATA_DEPRECATED) 2024-03-18T01:18:20.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:18:20 smithi029 ceph-mon[29331]: mds.? [v2:172.21.15.29:6828/1299751620,v1:172.21.15.29:6829/1299751620] up:reconnect 2024-03-18T01:18:20.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:18:20 smithi029 ceph-mon[29331]: from='client.? 172.21.15.29:0/2117714677' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "true", "yes_i_really_really_mean_it": true}]': finished 2024-03-18T01:18:20.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:18:20 smithi029 ceph-mon[29331]: mds.? [v2:172.21.15.90:6824/1998449484,v1:172.21.15.90:6825/1998449484] up:active 2024-03-18T01:18:20.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:18:20 smithi029 ceph-mon[29331]: fsmap cephfs:2/2 {0=cephfs.smithi029.brxqeh=up:reconnect,1=cephfs.smithi090.bohder=up:active} 1 up:standby-replay 1 up:standby 2024-03-18T01:18:20.776 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:18:20 smithi090 ceph-mon[36989]: pgmap v188: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.1 KiB/s rd, 1.6 KiB/s wr, 8 op/s 2024-03-18T01:18:20.776 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:18:20 smithi090 ceph-mon[36989]: Health check failed: 1 filesystem with deprecated feature inline_data (FS_INLINE_DATA_DEPRECATED) 2024-03-18T01:18:20.776 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:18:20 smithi090 ceph-mon[36989]: mds.? [v2:172.21.15.29:6828/1299751620,v1:172.21.15.29:6829/1299751620] up:reconnect 2024-03-18T01:18:20.776 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:18:20 smithi090 ceph-mon[36989]: from='client.? 172.21.15.29:0/2117714677' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "true", "yes_i_really_really_mean_it": true}]': finished 2024-03-18T01:18:20.776 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:18:20 smithi090 ceph-mon[36989]: mds.? [v2:172.21.15.90:6824/1998449484,v1:172.21.15.90:6825/1998449484] up:active 2024-03-18T01:18:20.776 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:18:20 smithi090 ceph-mon[36989]: fsmap cephfs:2/2 {0=cephfs.smithi029.brxqeh=up:reconnect,1=cephfs.smithi090.bohder=up:active} 1 up:standby-replay 1 up:standby 2024-03-18T01:18:21.414 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:18:21 smithi029 ceph-mon[29331]: mds.? [v2:172.21.15.29:6828/1299751620,v1:172.21.15.29:6829/1299751620] up:rejoin 2024-03-18T01:18:21.414 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:18:21 smithi029 ceph-mon[29331]: mds.? [v2:172.21.15.90:6826/439410941,v1:172.21.15.90:6827/439410941] up:standby-replay 2024-03-18T01:18:21.414 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:18:21 smithi029 ceph-mon[29331]: fsmap cephfs:2/2 {0=cephfs.smithi029.brxqeh=up:rejoin,1=cephfs.smithi090.bohder=up:active} 1 up:standby-replay 1 up:standby 2024-03-18T01:18:21.414 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:18:21 smithi029 ceph-mon[29331]: daemon mds.cephfs.smithi029.brxqeh is now active in filesystem cephfs as rank 0 2024-03-18T01:18:21.414 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:18:21 smithi029 ceph-mon[29331]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-18T01:18:21.414 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:18:21 smithi029 ceph-mon[29331]: mds.? [v2:172.21.15.29:6828/1299751620,v1:172.21.15.29:6829/1299751620] up:active 2024-03-18T01:18:21.415 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:18:21 smithi029 ceph-mon[29331]: fsmap cephfs:2 {0=cephfs.smithi029.brxqeh=up:active,1=cephfs.smithi090.bohder=up:active} 1 up:standby-replay 1 up:standby 2024-03-18T01:18:21.776 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:18:21 smithi090 ceph-mon[36989]: mds.? [v2:172.21.15.29:6828/1299751620,v1:172.21.15.29:6829/1299751620] up:rejoin 2024-03-18T01:18:21.776 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:18:21 smithi090 ceph-mon[36989]: mds.? [v2:172.21.15.90:6826/439410941,v1:172.21.15.90:6827/439410941] up:standby-replay 2024-03-18T01:18:21.776 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:18:21 smithi090 ceph-mon[36989]: fsmap cephfs:2/2 {0=cephfs.smithi029.brxqeh=up:rejoin,1=cephfs.smithi090.bohder=up:active} 1 up:standby-replay 1 up:standby 2024-03-18T01:18:21.776 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:18:21 smithi090 ceph-mon[36989]: daemon mds.cephfs.smithi029.brxqeh is now active in filesystem cephfs as rank 0 2024-03-18T01:18:21.776 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:18:21 smithi090 ceph-mon[36989]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-18T01:18:21.776 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:18:21 smithi090 ceph-mon[36989]: mds.? [v2:172.21.15.29:6828/1299751620,v1:172.21.15.29:6829/1299751620] up:active 2024-03-18T01:18:21.776 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:18:21 smithi090 ceph-mon[36989]: fsmap cephfs:2 {0=cephfs.smithi029.brxqeh=up:active,1=cephfs.smithi090.bohder=up:active} 1 up:standby-replay 1 up:standby 2024-03-18T01:18:22.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:18:22 smithi029 ceph-mon[29331]: pgmap v189: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 1.3 KiB/s wr, 7 op/s 2024-03-18T01:18:22.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:18:22 smithi029 ceph-mon[29331]: fsmap cephfs:2 {0=cephfs.smithi029.brxqeh=up:active,1=cephfs.smithi090.bohder=up:active} 2 up:standby-replay 2024-03-18T01:18:22.776 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:18:22 smithi090 ceph-mon[36989]: pgmap v189: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 1.3 KiB/s wr, 7 op/s 2024-03-18T01:18:22.776 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:18:22 smithi090 ceph-mon[36989]: fsmap cephfs:2 {0=cephfs.smithi029.brxqeh=up:active,1=cephfs.smithi090.bohder=up:active} 2 up:standby-replay 2024-03-18T01:18:23.214 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:23.211+0000 7f505cc27700 1 -- 172.21.15.29:0/2577681424 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f50580feda0 msgr2=0x7f50580faee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:18:23.214 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:23.211+0000 7f505cc27700 1 --2- 172.21.15.29:0/2577681424 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f50580feda0 0x7f50580faee0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f504c004940 tx=0x7f504c009e30 comp rx=0 tx=0).stop 2024-03-18T01:18:23.214 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:23.212+0000 7f505cc27700 1 -- 172.21.15.29:0/2577681424 shutdown_connections 2024-03-18T01:18:23.214 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:23.212+0000 7f505cc27700 1 --2- 172.21.15.29:0/2577681424 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f50580feda0 0x7f50580faee0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:18:23.215 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:23.212+0000 7f505cc27700 1 --2- 172.21.15.29:0/2577681424 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f50580fe450 0x7f50580fe860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:18:23.215 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:23.212+0000 7f505cc27700 1 -- 172.21.15.29:0/2577681424 >> 172.21.15.29:0/2577681424 conn(0x7f50580f6c10 msgr2=0x7f50580f9020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:18:23.215 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:23.212+0000 7f505cc27700 1 -- 172.21.15.29:0/2577681424 shutdown_connections 2024-03-18T01:18:23.215 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:23.212+0000 7f505cc27700 1 -- 172.21.15.29:0/2577681424 wait complete. 2024-03-18T01:18:23.215 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:23.212+0000 7f505cc27700 1 Processor -- start 2024-03-18T01:18:23.215 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:23.212+0000 7f505cc27700 1 -- start start 2024-03-18T01:18:23.215 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:23.212+0000 7f505cc27700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f50580fe450 0x7f505810d930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:18:23.215 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:23.212+0000 7f505cc27700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f50580feda0 0x7f505810de70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:18:23.215 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:23.212+0000 7f505cc27700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f505810e470 con 0x7f50580feda0 2024-03-18T01:18:23.215 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:23.212+0000 7f505cc27700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f505810e5e0 con 0x7f50580fe450 2024-03-18T01:18:23.215 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:23.212+0000 7f5055d9b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f50580feda0 0x7f505810de70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:18:23.215 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:23.212+0000 7f5055d9b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f50580feda0 0x7f505810de70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:58624/0 (socket says 172.21.15.29:58624) 2024-03-18T01:18:23.215 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:23.212+0000 7f5055d9b700 1 -- 172.21.15.29:0/3502236796 learned_addr learned my addr 172.21.15.29:0/3502236796 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-18T01:18:23.216 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:23.213+0000 7f505659c700 1 --2- 172.21.15.29:0/3502236796 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f50580fe450 0x7f505810d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:18:23.216 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:23.213+0000 7f5055d9b700 1 -- 172.21.15.29:0/3502236796 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f50580fe450 msgr2=0x7f505810d930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:18:23.216 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:23.213+0000 7f5055d9b700 1 --2- 172.21.15.29:0/3502236796 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f50580fe450 0x7f505810d930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:18:23.216 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:23.213+0000 7f5055d9b700 1 -- 172.21.15.29:0/3502236796 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f504c005040 con 0x7f50580feda0 2024-03-18T01:18:23.216 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:23.213+0000 7f5055d9b700 1 --2- 172.21.15.29:0/3502236796 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f50580feda0 0x7f505810de70 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f504c009150 tx=0x7f504c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:18:23.216 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:23.213+0000 7f5046ffd700 1 -- 172.21.15.29:0/3502236796 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f504c00e040 con 0x7f50580feda0 2024-03-18T01:18:23.216 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:23.213+0000 7f5046ffd700 1 -- 172.21.15.29:0/3502236796 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f504c00bcc0 con 0x7f50580feda0 2024-03-18T01:18:23.216 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:23.213+0000 7f5046ffd700 1 -- 172.21.15.29:0/3502236796 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f504c01c630 con 0x7f50580feda0 2024-03-18T01:18:23.216 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:23.213+0000 7f505cc27700 1 -- 172.21.15.29:0/3502236796 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5058112fe0 con 0x7f50580feda0 2024-03-18T01:18:23.216 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:23.213+0000 7f505cc27700 1 -- 172.21.15.29:0/3502236796 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5058113570 con 0x7f50580feda0 2024-03-18T01:18:23.217 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:23.214+0000 7f505cc27700 1 -- 172.21.15.29:0/3502236796 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f505804c710 con 0x7f50580feda0 2024-03-18T01:18:23.217 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:23.215+0000 7f5046ffd700 1 -- 172.21.15.29:0/3502236796 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f504c013420 con 0x7f50580feda0 2024-03-18T01:18:23.217 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:23.215+0000 7f5046ffd700 1 --2- 172.21.15.29:0/3502236796 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f50400692a0 0x7f504006b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:18:23.217 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:23.215+0000 7f5046ffd700 1 -- 172.21.15.29:0/3502236796 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f504c087e80 con 0x7f50580feda0 2024-03-18T01:18:23.217 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:23.215+0000 7f505659c700 1 --2- 172.21.15.29:0/3502236796 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f50400692a0 0x7f504006b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:18:23.217 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:23.215+0000 7f505659c700 1 --2- 172.21.15.29:0/3502236796 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f50400692a0 0x7f504006b750 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f5048007fd0 tx=0x7f5048009040 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:18:23.219 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:23.219+0000 7f5046ffd700 1 -- 172.21.15.29:0/3502236796 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175520 (secure 0 0 0) 0x7f504c057c90 con 0x7f50580feda0 2024-03-18T01:18:23.469 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:23.469+0000 7f505cc27700 1 -- 172.21.15.29:0/3502236796 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f5058105900 con 0x7f50580feda0 2024-03-18T01:18:23.469 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:23.469+0000 7f5046ffd700 1 -- 172.21.15.29:0/3502236796 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 20 v20) v1 ==== 76+0+1825 (secure 0 0 0) 0x7f504c057820 con 0x7f50580feda0 2024-03-18T01:18:23.470 INFO:teuthology.orchestra.run.smithi029.stdout:e20 2024-03-18T01:18:23.470 INFO:teuthology.orchestra.run.smithi029.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-18T01:18:23.470 INFO:teuthology.orchestra.run.smithi029.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-18T01:18:23.470 INFO:teuthology.orchestra.run.smithi029.stdout:legacy client fscid: 1 2024-03-18T01:18:23.470 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-18T01:18:23.470 INFO:teuthology.orchestra.run.smithi029.stdout:Filesystem 'cephfs' (1) 2024-03-18T01:18:23.470 INFO:teuthology.orchestra.run.smithi029.stdout:fs_name cephfs 2024-03-18T01:18:23.470 INFO:teuthology.orchestra.run.smithi029.stdout:epoch 20 2024-03-18T01:18:23.470 INFO:teuthology.orchestra.run.smithi029.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-03-18T01:18:23.470 INFO:teuthology.orchestra.run.smithi029.stdout:created 2024-03-18T01:18:03.228199+0000 2024-03-18T01:18:23.470 INFO:teuthology.orchestra.run.smithi029.stdout:modified 2024-03-18T01:18:21.293804+0000 2024-03-18T01:18:23.470 INFO:teuthology.orchestra.run.smithi029.stdout:tableserver 0 2024-03-18T01:18:23.470 INFO:teuthology.orchestra.run.smithi029.stdout:root 0 2024-03-18T01:18:23.470 INFO:teuthology.orchestra.run.smithi029.stdout:session_timeout 60 2024-03-18T01:18:23.470 INFO:teuthology.orchestra.run.smithi029.stdout:session_autoclose 300 2024-03-18T01:18:23.471 INFO:teuthology.orchestra.run.smithi029.stdout:max_file_size 1099511627776 2024-03-18T01:18:23.471 INFO:teuthology.orchestra.run.smithi029.stdout:required_client_features {} 2024-03-18T01:18:23.471 INFO:teuthology.orchestra.run.smithi029.stdout:last_failure 0 2024-03-18T01:18:23.471 INFO:teuthology.orchestra.run.smithi029.stdout:last_failure_osd_epoch 46 2024-03-18T01:18:23.471 INFO:teuthology.orchestra.run.smithi029.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-18T01:18:23.471 INFO:teuthology.orchestra.run.smithi029.stdout:max_mds 2 2024-03-18T01:18:23.471 INFO:teuthology.orchestra.run.smithi029.stdout:in 0,1 2024-03-18T01:18:23.471 INFO:teuthology.orchestra.run.smithi029.stdout:up {0=14480,1=24281} 2024-03-18T01:18:23.471 INFO:teuthology.orchestra.run.smithi029.stdout:failed 2024-03-18T01:18:23.471 INFO:teuthology.orchestra.run.smithi029.stdout:damaged 2024-03-18T01:18:23.471 INFO:teuthology.orchestra.run.smithi029.stdout:stopped 2024-03-18T01:18:23.471 INFO:teuthology.orchestra.run.smithi029.stdout:data_pools [3] 2024-03-18T01:18:23.471 INFO:teuthology.orchestra.run.smithi029.stdout:metadata_pool 2 2024-03-18T01:18:23.471 INFO:teuthology.orchestra.run.smithi029.stdout:inline_data enabled 2024-03-18T01:18:23.471 INFO:teuthology.orchestra.run.smithi029.stdout:balancer 2024-03-18T01:18:23.472 INFO:teuthology.orchestra.run.smithi029.stdout:standby_count_wanted 1 2024-03-18T01:18:23.472 INFO:teuthology.orchestra.run.smithi029.stdout:[mds.cephfs.smithi029.brxqeh{0:14480} state up:active seq 6 join_fscid=1 addr [v2:172.21.15.29:6828/1299751620,v1:172.21.15.29:6829/1299751620] compat {c=[1],r=[1],i=[7ff]}] 2024-03-18T01:18:23.472 INFO:teuthology.orchestra.run.smithi029.stdout:[mds.cephfs.smithi029.xuxvlr{0:14492} state up:standby-replay seq 1 join_fscid=1 addr [v2:172.21.15.29:6826/890079928,v1:172.21.15.29:6827/890079928] compat {c=[1],r=[1],i=[7ff]}] 2024-03-18T01:18:23.472 INFO:teuthology.orchestra.run.smithi029.stdout:[mds.cephfs.smithi090.bohder{1:24281} state up:active seq 4 join_fscid=1 addr [v2:172.21.15.90:6824/1998449484,v1:172.21.15.90:6825/1998449484] compat {c=[1],r=[1],i=[7ff]}] 2024-03-18T01:18:23.472 INFO:teuthology.orchestra.run.smithi029.stdout:[mds.cephfs.smithi090.trsnxz{1:24297} state up:standby-replay seq 2 join_fscid=1 addr [v2:172.21.15.90:6826/439410941,v1:172.21.15.90:6827/439410941] compat {c=[1],r=[1],i=[7ff]}] 2024-03-18T01:18:23.472 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-18T01:18:23.472 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-18T01:18:23.473 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:23.471+0000 7f505cc27700 1 -- 172.21.15.29:0/3502236796 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f50400692a0 msgr2=0x7f504006b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:18:23.473 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:23.471+0000 7f505cc27700 1 --2- 172.21.15.29:0/3502236796 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f50400692a0 0x7f504006b750 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f5048007fd0 tx=0x7f5048009040 comp rx=0 tx=0).stop 2024-03-18T01:18:23.473 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:23.471+0000 7f505cc27700 1 -- 172.21.15.29:0/3502236796 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f50580feda0 msgr2=0x7f505810de70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:18:23.473 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:23.471+0000 7f505cc27700 1 --2- 172.21.15.29:0/3502236796 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f50580feda0 0x7f505810de70 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f504c009150 tx=0x7f504c009e30 comp rx=0 tx=0).stop 2024-03-18T01:18:23.473 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:23.471+0000 7f505cc27700 1 -- 172.21.15.29:0/3502236796 shutdown_connections 2024-03-18T01:18:23.473 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:23.471+0000 7f505cc27700 1 --2- 172.21.15.29:0/3502236796 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f50400692a0 0x7f504006b750 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:18:23.473 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:23.471+0000 7f505cc27700 1 --2- 172.21.15.29:0/3502236796 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f50580fe450 0x7f505810d930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:18:23.473 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:23.471+0000 7f505cc27700 1 --2- 172.21.15.29:0/3502236796 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f50580feda0 0x7f505810de70 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:18:23.473 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:23.471+0000 7f505cc27700 1 -- 172.21.15.29:0/3502236796 >> 172.21.15.29:0/3502236796 conn(0x7f50580f6c10 msgr2=0x7f50580f8290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:18:23.473 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:23.472+0000 7f505cc27700 1 -- 172.21.15.29:0/3502236796 shutdown_connections 2024-03-18T01:18:23.473 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:23.472+0000 7f505cc27700 1 -- 172.21.15.29:0/3502236796 wait complete. 2024-03-18T01:18:23.473 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 20 2024-03-18T01:18:24.137 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 37f51182-e4c4-11ee-95c9-87774f69a715 -- bash -c 'ceph --format=json fs dump | jq -e ".filesystems | length == 1"' 2024-03-18T01:18:24.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:18:24 smithi090 ceph-mon[36989]: pgmap v190: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 3.3 KiB/s rd, 102 B/s wr, 6 op/s 2024-03-18T01:18:24.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:18:24 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:18:24.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:18:24 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:18:24.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:18:24 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:18:24.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:18:24 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T01:18:24.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:18:24 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:18:24.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:18:24 smithi090 ceph-mon[36989]: from='client.? 172.21.15.29:0/3502236796' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-18T01:18:24.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:18:24 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T01:18:24.558 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:18:24 smithi029 ceph-mon[29331]: pgmap v190: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 3.3 KiB/s rd, 102 B/s wr, 6 op/s 2024-03-18T01:18:24.558 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:18:24 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:18:24.558 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:18:24 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:18:24.558 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:18:24 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:18:24.558 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:18:24 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T01:18:24.558 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:18:24 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:18:24.559 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:18:24 smithi029 ceph-mon[29331]: from='client.? 172.21.15.29:0/3502236796' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-18T01:18:24.559 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:18:24 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T01:18:25.146 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/37f51182-e4c4-11ee-95c9-87774f69a715/mon.smithi029/config 2024-03-18T01:18:26.525 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:18:26 smithi029 ceph-mon[29331]: pgmap v191: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 6.4 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-18T01:18:26.525 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:18:26 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:18:26.525 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:18:26 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:18:26.525 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:18:26 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:18:26.526 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:18:26 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-18T01:18:26.776 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:18:26 smithi090 ceph-mon[36989]: pgmap v191: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 6.4 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-18T01:18:26.776 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:18:26 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:18:26.776 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:18:26 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:18:26.776 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:18:26 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:18:26.776 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:18:26 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-18T01:18:27.045 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:26.936+0000 7f360c758700 1 -- 172.21.15.29:0/2002087731 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f36040fce40 msgr2=0x7f36040fd250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:18:27.045 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:26.936+0000 7f360c758700 1 --2- 172.21.15.29:0/2002087731 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f36040fce40 0x7f36040fd250 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f35fc004940 tx=0x7f35fc009e30 comp rx=0 tx=0).stop 2024-03-18T01:18:27.045 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:26.936+0000 7f360c758700 1 -- 172.21.15.29:0/2002087731 shutdown_connections 2024-03-18T01:18:27.045 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:26.936+0000 7f360c758700 1 --2- 172.21.15.29:0/2002087731 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f36040fe040 0x7f36040fe4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:18:27.045 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:26.936+0000 7f360c758700 1 --2- 172.21.15.29:0/2002087731 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f36040fce40 0x7f36040fd250 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:18:27.045 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:26.936+0000 7f360c758700 1 -- 172.21.15.29:0/2002087731 >> 172.21.15.29:0/2002087731 conn(0x7f36040f8570 msgr2=0x7f36040fa980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:18:27.045 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:26.936+0000 7f360c758700 1 -- 172.21.15.29:0/2002087731 shutdown_connections 2024-03-18T01:18:27.046 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:26.936+0000 7f360c758700 1 -- 172.21.15.29:0/2002087731 wait complete. 2024-03-18T01:18:27.046 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:26.937+0000 7f360c758700 1 Processor -- start 2024-03-18T01:18:27.046 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:26.937+0000 7f360c758700 1 -- start start 2024-03-18T01:18:27.046 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:26.937+0000 7f360c758700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f36040fce40 0x7f360410da30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:18:27.046 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:26.937+0000 7f360c758700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f36040fe040 0x7f360410df70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:18:27.046 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:26.937+0000 7f360c758700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f360410e570 con 0x7f36040fce40 2024-03-18T01:18:27.046 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:26.937+0000 7f360c758700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f360410edc0 con 0x7f36040fe040 2024-03-18T01:18:27.046 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:26.937+0000 7f360a4f4700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f36040fce40 0x7f360410da30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:18:27.046 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:26.937+0000 7f360a4f4700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f36040fce40 0x7f360410da30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:58636/0 (socket says 172.21.15.29:58636) 2024-03-18T01:18:27.046 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:26.937+0000 7f360a4f4700 1 -- 172.21.15.29:0/3245591007 learned_addr learned my addr 172.21.15.29:0/3245591007 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-18T01:18:27.046 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:26.937+0000 7f3609cf3700 1 --2- 172.21.15.29:0/3245591007 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f36040fe040 0x7f360410df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:18:27.046 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:26.937+0000 7f360a4f4700 1 -- 172.21.15.29:0/3245591007 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f36040fe040 msgr2=0x7f360410df70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:18:27.046 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:26.937+0000 7f360a4f4700 1 --2- 172.21.15.29:0/3245591007 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f36040fe040 0x7f360410df70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:18:27.046 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:26.937+0000 7f360a4f4700 1 -- 172.21.15.29:0/3245591007 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35fc005040 con 0x7f36040fce40 2024-03-18T01:18:27.047 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:26.937+0000 7f360a4f4700 1 --2- 172.21.15.29:0/3245591007 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f36040fce40 0x7f360410da30 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f35fc00b3b0 tx=0x7f35fc009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:18:27.047 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:26.937+0000 7f3602ffd700 1 -- 172.21.15.29:0/3245591007 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f35fc00e040 con 0x7f36040fce40 2024-03-18T01:18:27.047 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:26.937+0000 7f360c758700 1 -- 172.21.15.29:0/3245591007 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f360410f040 con 0x7f36040fce40 2024-03-18T01:18:27.047 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:26.937+0000 7f3602ffd700 1 -- 172.21.15.29:0/3245591007 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f35fc00bcc0 con 0x7f36040fce40 2024-03-18T01:18:27.047 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:26.938+0000 7f3602ffd700 1 -- 172.21.15.29:0/3245591007 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f35fc01c630 con 0x7f36040fce40 2024-03-18T01:18:27.047 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:26.938+0000 7f360c758700 1 -- 172.21.15.29:0/3245591007 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f360410f580 con 0x7f36040fce40 2024-03-18T01:18:27.047 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:26.938+0000 7f360c758700 1 -- 172.21.15.29:0/3245591007 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f360405f5f0 con 0x7f36040fce40 2024-03-18T01:18:27.047 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:26.939+0000 7f3602ffd700 1 -- 172.21.15.29:0/3245591007 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f35fc0139f0 con 0x7f36040fce40 2024-03-18T01:18:27.047 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:26.939+0000 7f3602ffd700 1 --2- 172.21.15.29:0/3245591007 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f35ec0692a0 0x7f35ec06b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:18:27.047 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:26.939+0000 7f3602ffd700 1 -- 172.21.15.29:0/3245591007 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f35fc087fc0 con 0x7f36040fce40 2024-03-18T01:18:27.047 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:26.939+0000 7f3609cf3700 1 --2- 172.21.15.29:0/3245591007 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f35ec0692a0 0x7f35ec06b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:18:27.047 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:26.940+0000 7f3609cf3700 1 --2- 172.21.15.29:0/3245591007 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f35ec0692a0 0x7f35ec06b750 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f35f8004d60 tx=0x7f35f800b040 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:18:27.047 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:26.945+0000 7f3602ffd700 1 -- 172.21.15.29:0/3245591007 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175520 (secure 0 0 0) 0x7f35fc057d30 con 0x7f36040fce40 2024-03-18T01:18:27.182 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:27.181+0000 7f360c758700 1 -- 172.21.15.29:0/3245591007 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f360404bed0 con 0x7f36040fce40 2024-03-18T01:18:27.182 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:27.182+0000 7f3602ffd700 1 -- 172.21.15.29:0/3245591007 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) v1 ==== 94+0+4713 (secure 0 0 0) 0x7f35fc0578c0 con 0x7f36040fce40 2024-03-18T01:18:27.184 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:27.184+0000 7f360c758700 1 -- 172.21.15.29:0/3245591007 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f35ec0692a0 msgr2=0x7f35ec06b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:18:27.184 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:27.184+0000 7f360c758700 1 --2- 172.21.15.29:0/3245591007 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f35ec0692a0 0x7f35ec06b750 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f35f8004d60 tx=0x7f35f800b040 comp rx=0 tx=0).stop 2024-03-18T01:18:27.184 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:27.184+0000 7f360c758700 1 -- 172.21.15.29:0/3245591007 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f36040fce40 msgr2=0x7f360410da30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:18:27.185 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:27.184+0000 7f360c758700 1 --2- 172.21.15.29:0/3245591007 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f36040fce40 0x7f360410da30 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f35fc00b3b0 tx=0x7f35fc009e30 comp rx=0 tx=0).stop 2024-03-18T01:18:27.185 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:27.184+0000 7f360c758700 1 -- 172.21.15.29:0/3245591007 shutdown_connections 2024-03-18T01:18:27.185 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:27.184+0000 7f360c758700 1 --2- 172.21.15.29:0/3245591007 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f35ec0692a0 0x7f35ec06b750 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:18:27.185 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:27.184+0000 7f360c758700 1 --2- 172.21.15.29:0/3245591007 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f36040fce40 0x7f360410da30 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:18:27.185 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:27.184+0000 7f360c758700 1 --2- 172.21.15.29:0/3245591007 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f36040fe040 0x7f360410df70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:18:27.185 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:27.184+0000 7f360c758700 1 -- 172.21.15.29:0/3245591007 >> 172.21.15.29:0/3245591007 conn(0x7f36040f8570 msgr2=0x7f36040f9c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:18:27.185 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:27.184+0000 7f360c758700 1 -- 172.21.15.29:0/3245591007 shutdown_connections 2024-03-18T01:18:27.185 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:27.184+0000 7f360c758700 1 -- 172.21.15.29:0/3245591007 wait complete. 2024-03-18T01:18:27.185 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 20 2024-03-18T01:18:27.194 INFO:teuthology.orchestra.run.smithi029.stdout:true 2024-03-18T01:18:27.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:18:27 smithi029 ceph-mon[29331]: from='client.? 172.21.15.29:0/3245591007' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T01:18:27.776 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:18:27 smithi090 ceph-mon[36989]: from='client.? 172.21.15.29:0/3245591007' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T01:18:28.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:18:28 smithi029 ceph-mon[29331]: pgmap v192: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 7.5 KiB/s rd, 0 B/s wr, 9 op/s 2024-03-18T01:18:28.673 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 37f51182-e4c4-11ee-95c9-87774f69a715 -- bash -c 'while ! ceph --format=json mds versions | jq -e ". | add == 4"; do sleep 1; done' 2024-03-18T01:18:28.776 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:18:28 smithi090 ceph-mon[36989]: pgmap v192: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 7.5 KiB/s rd, 0 B/s wr, 9 op/s 2024-03-18T01:18:29.426 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/37f51182-e4c4-11ee-95c9-87774f69a715/mon.smithi029/config 2024-03-18T01:18:30.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:18:30 smithi029 ceph-mon[29331]: pgmap v193: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 6.8 KiB/s rd, 0 B/s wr, 8 op/s 2024-03-18T01:18:30.776 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:18:30 smithi090 ceph-mon[36989]: pgmap v193: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 6.8 KiB/s rd, 0 B/s wr, 8 op/s 2024-03-18T01:18:31.695 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:31.695+0000 7f93efe26700 1 -- 172.21.15.29:0/1976332817 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f93e80fdfe0 msgr2=0x7f93e80fe450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:18:31.696 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:31.695+0000 7f93efe26700 1 --2- 172.21.15.29:0/1976332817 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f93e80fdfe0 0x7f93e80fe450 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f93dc004940 tx=0x7f93dc009e30 comp rx=0 tx=0).stop 2024-03-18T01:18:31.696 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:31.695+0000 7f93efe26700 1 -- 172.21.15.29:0/1976332817 shutdown_connections 2024-03-18T01:18:31.696 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:31.695+0000 7f93efe26700 1 --2- 172.21.15.29:0/1976332817 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f93e80fdfe0 0x7f93e80fe450 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:18:31.696 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:31.695+0000 7f93efe26700 1 --2- 172.21.15.29:0/1976332817 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f93e80fce70 0x7f93e80fd280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:18:31.696 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:31.695+0000 7f93efe26700 1 -- 172.21.15.29:0/1976332817 >> 172.21.15.29:0/1976332817 conn(0x7f93e80f8510 msgr2=0x7f93e80fa960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:18:31.696 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:31.695+0000 7f93efe26700 1 -- 172.21.15.29:0/1976332817 shutdown_connections 2024-03-18T01:18:31.696 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:31.695+0000 7f93efe26700 1 -- 172.21.15.29:0/1976332817 wait complete. 2024-03-18T01:18:31.697 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:31.696+0000 7f93efe26700 1 Processor -- start 2024-03-18T01:18:31.697 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:31.696+0000 7f93efe26700 1 -- start start 2024-03-18T01:18:31.698 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:31.696+0000 7f93efe26700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f93e80fce70 0x7f93e810da00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:18:31.698 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:31.696+0000 7f93efe26700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f93e80fdfe0 0x7f93e810df40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:18:31.698 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:31.696+0000 7f93efe26700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93e810e540 con 0x7f93e80fce70 2024-03-18T01:18:31.698 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:31.696+0000 7f93efe26700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93e810ed90 con 0x7f93e80fdfe0 2024-03-18T01:18:31.698 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:31.696+0000 7f93edbc2700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f93e80fce70 0x7f93e810da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:18:31.698 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:31.696+0000 7f93edbc2700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f93e80fce70 0x7f93e810da00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:58654/0 (socket says 172.21.15.29:58654) 2024-03-18T01:18:31.698 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:31.696+0000 7f93edbc2700 1 -- 172.21.15.29:0/3478543008 learned_addr learned my addr 172.21.15.29:0/3478543008 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-18T01:18:31.698 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:31.696+0000 7f93ed3c1700 1 --2- 172.21.15.29:0/3478543008 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f93e80fdfe0 0x7f93e810df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:18:31.698 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:31.696+0000 7f93edbc2700 1 -- 172.21.15.29:0/3478543008 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f93e80fdfe0 msgr2=0x7f93e810df40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:18:31.698 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:31.696+0000 7f93edbc2700 1 --2- 172.21.15.29:0/3478543008 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f93e80fdfe0 0x7f93e810df40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:18:31.698 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:31.696+0000 7f93edbc2700 1 -- 172.21.15.29:0/3478543008 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93dc005040 con 0x7f93e80fce70 2024-03-18T01:18:31.698 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:31.696+0000 7f93ed3c1700 1 --2- 172.21.15.29:0/3478543008 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f93e80fdfe0 0x7f93e810df40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T01:18:31.698 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:31.696+0000 7f93edbc2700 1 --2- 172.21.15.29:0/3478543008 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f93e80fce70 0x7f93e810da00 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f93d800cac0 tx=0x7f93d8007ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:18:31.699 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:31.697+0000 7f93e67fc700 1 -- 172.21.15.29:0/3478543008 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f93d800aa50 con 0x7f93e80fce70 2024-03-18T01:18:31.699 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:31.697+0000 7f93e67fc700 1 -- 172.21.15.29:0/3478543008 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f93d800abb0 con 0x7f93e80fce70 2024-03-18T01:18:31.699 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:31.697+0000 7f93efe26700 1 -- 172.21.15.29:0/3478543008 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93e810f070 con 0x7f93e80fce70 2024-03-18T01:18:31.699 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:31.697+0000 7f93e67fc700 1 -- 172.21.15.29:0/3478543008 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f93d80115e0 con 0x7f93e80fce70 2024-03-18T01:18:31.699 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:31.697+0000 7f93efe26700 1 -- 172.21.15.29:0/3478543008 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93e810f5b0 con 0x7f93e80fce70 2024-03-18T01:18:31.700 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:31.698+0000 7f93efe26700 1 -- 172.21.15.29:0/3478543008 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93e805f5f0 con 0x7f93e80fce70 2024-03-18T01:18:31.700 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:31.698+0000 7f93e67fc700 1 -- 172.21.15.29:0/3478543008 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f93d800ad20 con 0x7f93e80fce70 2024-03-18T01:18:31.700 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:31.698+0000 7f93e67fc700 1 --2- 172.21.15.29:0/3478543008 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f93d00692a0 0x7f93d006b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:18:31.700 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:31.699+0000 7f93e67fc700 1 -- 172.21.15.29:0/3478543008 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f93d8087970 con 0x7f93e80fce70 2024-03-18T01:18:31.700 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:31.699+0000 7f93ed3c1700 1 --2- 172.21.15.29:0/3478543008 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f93d00692a0 0x7f93d006b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:18:31.700 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:31.699+0000 7f93ed3c1700 1 --2- 172.21.15.29:0/3478543008 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f93d00692a0 0x7f93d006b750 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f93dc00b3b0 tx=0x7f93dc009e30 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:18:31.703 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:31.703+0000 7f93e67fc700 1 -- 172.21.15.29:0/3478543008 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175520 (secure 0 0 0) 0x7f93d8057660 con 0x7f93e80fce70 2024-03-18T01:18:31.989 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:31.989+0000 7f93efe26700 1 -- 172.21.15.29:0/3478543008 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mds versions", "format": "json"} v 0) v1 -- 0x7f93e804bed0 con 0x7f93e80fce70 2024-03-18T01:18:31.989 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:31.989+0000 7f93e67fc700 1 -- 172.21.15.29:0/3478543008 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mds versions", "format": "json"}]=0 v20) v1 ==== 78+0+99 (secure 0 0 0) 0x7f93d80571f0 con 0x7f93e80fce70 2024-03-18T01:18:31.992 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:31.992+0000 7f93efe26700 1 -- 172.21.15.29:0/3478543008 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f93d00692a0 msgr2=0x7f93d006b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:18:31.992 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:31.992+0000 7f93efe26700 1 --2- 172.21.15.29:0/3478543008 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f93d00692a0 0x7f93d006b750 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f93dc00b3b0 tx=0x7f93dc009e30 comp rx=0 tx=0).stop 2024-03-18T01:18:31.992 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:31.992+0000 7f93efe26700 1 -- 172.21.15.29:0/3478543008 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f93e80fce70 msgr2=0x7f93e810da00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:18:31.992 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:31.992+0000 7f93efe26700 1 --2- 172.21.15.29:0/3478543008 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f93e80fce70 0x7f93e810da00 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f93d800cac0 tx=0x7f93d8007ee0 comp rx=0 tx=0).stop 2024-03-18T01:18:31.992 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:31.992+0000 7f93efe26700 1 -- 172.21.15.29:0/3478543008 shutdown_connections 2024-03-18T01:18:31.992 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:31.992+0000 7f93efe26700 1 --2- 172.21.15.29:0/3478543008 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f93d00692a0 0x7f93d006b750 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:18:31.992 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:31.992+0000 7f93efe26700 1 --2- 172.21.15.29:0/3478543008 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f93e80fce70 0x7f93e810da00 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:18:31.992 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:31.992+0000 7f93efe26700 1 --2- 172.21.15.29:0/3478543008 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f93e80fdfe0 0x7f93e810df40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:18:31.992 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:31.992+0000 7f93efe26700 1 -- 172.21.15.29:0/3478543008 >> 172.21.15.29:0/3478543008 conn(0x7f93e80f8510 msgr2=0x7f93e80f9bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:18:31.993 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:31.992+0000 7f93efe26700 1 -- 172.21.15.29:0/3478543008 shutdown_connections 2024-03-18T01:18:31.993 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:31.992+0000 7f93efe26700 1 -- 172.21.15.29:0/3478543008 wait complete. 2024-03-18T01:18:32.002 INFO:teuthology.orchestra.run.smithi029.stdout:true 2024-03-18T01:18:32.662 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:18:32 smithi029 ceph-mon[29331]: pgmap v194: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 6.5 KiB/s rd, 0 B/s wr, 8 op/s 2024-03-18T01:18:32.662 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:18:32 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:18:32.662 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:18:32 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:18:32.662 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:18:32 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:18:32.662 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:18:32 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T01:18:32.662 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:18:32 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:18:32.662 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:18:32 smithi029 ceph-mon[29331]: from='client.? 172.21.15.29:0/3478543008' entity='client.admin' cmd=[{"prefix": "mds versions", "format": "json"}]: dispatch 2024-03-18T01:18:32.691 INFO:teuthology.run_tasks:Running task fs.pre_upgrade_save... 2024-03-18T01:18:32.703 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T01:18:32.776 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:18:32 smithi090 ceph-mon[36989]: pgmap v194: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 6.5 KiB/s rd, 0 B/s wr, 8 op/s 2024-03-18T01:18:32.776 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:18:32 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:18:32.776 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:18:32 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:18:32.776 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:18:32 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:18:32.776 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:18:32 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T01:18:32.777 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:18:32 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:18:32.777 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:18:32 smithi090 ceph-mon[36989]: from='client.? 172.21.15.29:0/3478543008' entity='client.admin' cmd=[{"prefix": "mds versions", "format": "json"}]: dispatch 2024-03-18T01:18:32.817 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:32.815+0000 7f6f76eae640 1 -- 172.21.15.29:0/870661496 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6f70065dc0 msgr2=0x7f6f700fee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:18:32.817 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:32.815+0000 7f6f76eae640 1 --2- 172.21.15.29:0/870661496 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6f70065dc0 0x7f6f700fee20 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f6f68009c50 tx=0x7f6f68004d10 comp rx=0 tx=0).stop 2024-03-18T01:18:32.817 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:32.815+0000 7f6f76eae640 1 -- 172.21.15.29:0/870661496 shutdown_connections 2024-03-18T01:18:32.817 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:32.815+0000 7f6f76eae640 1 --2- 172.21.15.29:0/870661496 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6f700ff360 0x7f6f70101740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:18:32.818 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:32.815+0000 7f6f76eae640 1 --2- 172.21.15.29:0/870661496 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6f70065dc0 0x7f6f700fee20 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:18:32.818 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:32.815+0000 7f6f76eae640 1 -- 172.21.15.29:0/870661496 >> 172.21.15.29:0/870661496 conn(0x7f6f700f6870 msgr2=0x7f6f700f8ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:18:32.818 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:32.815+0000 7f6f76eae640 1 -- 172.21.15.29:0/870661496 shutdown_connections 2024-03-18T01:18:32.818 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:32.815+0000 7f6f76eae640 1 -- 172.21.15.29:0/870661496 wait complete. 2024-03-18T01:18:32.818 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:32.816+0000 7f6f76eae640 1 Processor -- start 2024-03-18T01:18:32.818 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:32.816+0000 7f6f76eae640 1 -- start start 2024-03-18T01:18:32.818 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:32.816+0000 7f6f76eae640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6f70065dc0 0x7f6f7010ccc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:18:32.818 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:32.816+0000 7f6f76eae640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6f700ff360 0x7f6f7010d200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:18:32.818 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:32.816+0000 7f6f76eae640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f7010d880 con 0x7f6f700ff360 2024-03-18T01:18:32.818 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:32.816+0000 7f6f76eae640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f70112240 con 0x7f6f70065dc0 2024-03-18T01:18:32.818 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:32.816+0000 7f6f6ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6f700ff360 0x7f6f7010d200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:18:32.818 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:32.816+0000 7f6f6ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6f700ff360 0x7f6f7010d200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:55180/0 (socket says 172.21.15.29:55180) 2024-03-18T01:18:32.818 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:32.816+0000 7f6f6ffff640 1 -- 172.21.15.29:0/1351658686 learned_addr learned my addr 172.21.15.29:0/1351658686 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-18T01:18:32.818 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:32.816+0000 7f6f74c4a640 1 --2- 172.21.15.29:0/1351658686 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6f70065dc0 0x7f6f7010ccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:18:32.819 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:32.816+0000 7f6f6ffff640 1 -- 172.21.15.29:0/1351658686 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6f70065dc0 msgr2=0x7f6f7010ccc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:18:32.819 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:32.816+0000 7f6f6ffff640 1 --2- 172.21.15.29:0/1351658686 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6f70065dc0 0x7f6f7010ccc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:18:32.819 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:32.816+0000 7f6f6ffff640 1 -- 172.21.15.29:0/1351658686 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f680098b0 con 0x7f6f700ff360 2024-03-18T01:18:32.819 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:32.817+0000 7f6f6ffff640 1 --2- 172.21.15.29:0/1351658686 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6f700ff360 0x7f6f7010d200 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f6f6400c860 tx=0x7f6f64004430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:18:32.819 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:32.817+0000 7f6f6d7fa640 1 -- 172.21.15.29:0/1351658686 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6f6400aa30 con 0x7f6f700ff360 2024-03-18T01:18:32.819 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:32.817+0000 7f6f6d7fa640 1 -- 172.21.15.29:0/1351658686 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6f6400ce50 con 0x7f6f700ff360 2024-03-18T01:18:32.819 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:32.817+0000 7f6f76eae640 1 -- 172.21.15.29:0/1351658686 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f701123e0 con 0x7f6f700ff360 2024-03-18T01:18:32.819 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:32.817+0000 7f6f6d7fa640 1 -- 172.21.15.29:0/1351658686 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6f640115e0 con 0x7f6f700ff360 2024-03-18T01:18:32.819 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:32.817+0000 7f6f76eae640 1 -- 172.21.15.29:0/1351658686 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f70112950 con 0x7f6f700ff360 2024-03-18T01:18:32.820 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:32.818+0000 7f6f76eae640 1 -- 172.21.15.29:0/1351658686 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f7004bcc0 con 0x7f6f700ff360 2024-03-18T01:18:32.820 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:32.818+0000 7f6f6d7fa640 1 -- 172.21.15.29:0/1351658686 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f6f6400ab90 con 0x7f6f700ff360 2024-03-18T01:18:32.820 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:32.819+0000 7f6f6d7fa640 1 --2- 172.21.15.29:0/1351658686 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f6f58069210 0x7f6f5806b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:18:32.820 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:32.819+0000 7f6f6d7fa640 1 -- 172.21.15.29:0/1351658686 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f6f64087420 con 0x7f6f700ff360 2024-03-18T01:18:32.820 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:32.819+0000 7f6f74c4a640 1 --2- 172.21.15.29:0/1351658686 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f6f58069210 0x7f6f5806b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:18:32.820 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:32.819+0000 7f6f74c4a640 1 --2- 172.21.15.29:0/1351658686 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f6f58069210 0x7f6f5806b6c0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f6f68004a30 tx=0x7f6f6802e040 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:18:32.823 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:32.823+0000 7f6f6d7fa640 1 -- 172.21.15.29:0/1351658686 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175520 (secure 0 0 0) 0x7f6f64053700 con 0x7f6f700ff360 2024-03-18T01:18:32.997 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:32.996+0000 7f6f76eae640 1 -- 172.21.15.29:0/1351658686 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6f70104d10 con 0x7f6f700ff360 2024-03-18T01:18:32.998 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:32.997+0000 7f6f6d7fa640 1 -- 172.21.15.29:0/1351658686 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v46) v1 ==== 74+0+14350 (secure 0 0 0) 0x7f6f6400ae40 con 0x7f6f700ff360 2024-03-18T01:18:32.998 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-18T01:18:32.998 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":46,"fsid":"37f51182-e4c4-11ee-95c9-87774f69a715","created":"2024-03-18T01:10:49.947064+0000","modified":"2024-03-18T01:18:17.281965+0000","last_up_change":"2024-03-18T01:16:41.540245+0000","last_in_change":"2024-03-18T01:16:23.078422+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T01:15:31.263727+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"cephfs.cephfs.meta","create_time":"2024-03-18T01:18:01.725916+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"44","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs.cephfs.data","create_time":"2024-03-18T01:18:02.468256+0000","flags":131073,"flags_names":"hashpspool,bulk","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"45","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"decb820d-5c65-4317-b3f0-46522f1d1326","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":895619893},{"type":"v1","addr":"172.21.15.29:6803","nonce":895619893}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":895619893},{"type":"v1","addr":"172.21.15.29:6805","nonce":895619893}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":895619893},{"type":"v1","addr":"172.21.15.29:6809","nonce":895619893}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":895619893},{"type":"v1","addr":"172.21.15.29:6807","nonce":895619893}]},"public_addr":"172.21.15.29:6803/895619893","cluster_addr":"172.21.15.29:6805/895619893","heartbeat_back_addr":"172.21.15.29:6809/895619893","heartbeat_front_addr":"172.21.15.29:6807/895619893","state":["exists","up"]},{"osd":1,"uuid":"3b1f0295-4dd6-4bb1-9847-93bd98677db6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":2615748531},{"type":"v1","addr":"172.21.15.29:6811","nonce":2615748531}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":2615748531},{"type":"v1","addr":"172.21.15.29:6813","nonce":2615748531}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":2615748531},{"type":"v1","addr":"172.21.15.29:6817","nonce":2615748531}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":2615748531},{"type":"v1","addr":"172.21.15.29:6815","nonce":2615748531}]},"public_addr":"172.21.15.29:6811/2615748531","cluster_addr":"172.21.15.29:6813/2615748531","heartbeat_back_addr":"172.21.15.29:6817/2615748531","heartbeat_front_addr":"172.21.15.29:6815/2615748531","state":["exists","up"]},{"osd":2,"uuid":"826d30cd-7b7d-4f93-9567-c6bf6120dfe0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":576060095},{"type":"v1","addr":"172.21.15.29:6819","nonce":576060095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6820","nonce":576060095},{"type":"v1","addr":"172.21.15.29:6821","nonce":576060095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":576060095},{"type":"v1","addr":"172.21.15.29:6825","nonce":576060095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":576060095},{"type":"v1","addr":"172.21.15.29:6823","nonce":576060095}]},"public_addr":"172.21.15.29:6819/576060095","cluster_addr":"172.21.15.29:6821/576060095","heartbeat_back_addr":"172.21.15.29:6825/576060095","heartbeat_front_addr":"172.21.15.29:6823/576060095","state":["exists","up"]},{"osd":3,"uuid":"cba371da-b198-4b14-a3ac-fcf90d1a91e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":1797049062},{"type":"v1","addr":"172.21.15.90:6801","nonce":1797049062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":1797049062},{"type":"v1","addr":"172.21.15.90:6803","nonce":1797049062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":1797049062},{"type":"v1","addr":"172.21.15.90:6807","nonce":1797049062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":1797049062},{"type":"v1","addr":"172.21.15.90:6805","nonce":1797049062}]},"public_addr":"172.21.15.90:6801/1797049062","cluster_addr":"172.21.15.90:6803/1797049062","heartbeat_back_addr":"172.21.15.90:6807/1797049062","heartbeat_front_addr":"172.21.15.90:6805/1797049062","state":["exists","up"]},{"osd":4,"uuid":"c18771da-2efb-4897-a07a-ce48c94e2dee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":2675255441},{"type":"v1","addr":"172.21.15.90:6809","nonce":2675255441}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":2675255441},{"type":"v1","addr":"172.21.15.90:6811","nonce":2675255441}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":2675255441},{"type":"v1","addr":"172.21.15.90:6815","nonce":2675255441}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":2675255441},{"type":"v1","addr":"172.21.15.90:6813","nonce":2675255441}]},"public_addr":"172.21.15.90:6809/2675255441","cluster_addr":"172.21.15.90:6811/2675255441","heartbeat_back_addr":"172.21.15.90:6815/2675255441","heartbeat_front_addr":"172.21.15.90:6813/2675255441","state":["exists","up"]},{"osd":5,"uuid":"82248e69-74f9-4f46-a3fd-9224dcc8eae6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":2630668390},{"type":"v1","addr":"172.21.15.90:6817","nonce":2630668390}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6818","nonce":2630668390},{"type":"v1","addr":"172.21.15.90:6819","nonce":2630668390}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":2630668390},{"type":"v1","addr":"172.21.15.90:6823","nonce":2630668390}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6820","nonce":2630668390},{"type":"v1","addr":"172.21.15.90:6821","nonce":2630668390}]},"public_addr":"172.21.15.90:6817/2630668390","cluster_addr":"172.21.15.90:6819/2630668390","heartbeat_back_addr":"172.21.15.90:6823/2630668390","heartbeat_front_addr":"172.21.15.90:6821/2630668390","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:14:30.758603+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:14:59.452381+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:15:28.377239+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:15:51.151100+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:16:15.663658+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:16:40.030041+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.29:6827/1784211444":"2024-03-19T01:18:17.281934+0000","172.21.15.29:0/3826687028":"2024-03-19T01:11:59.169888+0000","172.21.15.29:6800/950856841":"2024-03-19T01:11:59.169888+0000","172.21.15.29:0/1016951963":"2024-03-19T01:11:26.433683+0000","172.21.15.29:0/4093790008":"2024-03-19T01:11:26.433683+0000","172.21.15.29:6800/1702596206":"2024-03-19T01:11:26.433683+0000","172.21.15.29:0/3191846431":"2024-03-19T01:13:10.926615+0000","172.21.15.29:6800/2687730237":"2024-03-19T01:13:10.926615+0000","172.21.15.29:0/2614521832":"2024-03-19T01:11:26.433683+0000","172.21.15.29:6801/1702596206":"2024-03-19T01:11:26.433683+0000","172.21.15.29:6826/1784211444":"2024-03-19T01:18:17.281934+0000","172.21.15.29:6801/950856841":"2024-03-19T01:11:59.169888+0000","172.21.15.29:0/3604736621":"2024-03-19T01:11:59.169888+0000","172.21.15.29:0/3723093724":"2024-03-19T01:11:59.169888+0000","172.21.15.29:6801/2687730237":"2024-03-19T01:13:10.926615+0000","172.21.15.29:0/3466320979":"2024-03-19T01:13:10.926615+0000","172.21.15.29:0/2762171838":"2024-03-19T01:13:10.926615+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T01:18:33.000 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:32.999+0000 7f6f76eae640 1 -- 172.21.15.29:0/1351658686 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f6f58069210 msgr2=0x7f6f5806b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:18:33.000 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:32.999+0000 7f6f76eae640 1 --2- 172.21.15.29:0/1351658686 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f6f58069210 0x7f6f5806b6c0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f6f68004a30 tx=0x7f6f6802e040 comp rx=0 tx=0).stop 2024-03-18T01:18:33.000 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:32.999+0000 7f6f76eae640 1 -- 172.21.15.29:0/1351658686 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6f700ff360 msgr2=0x7f6f7010d200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:18:33.000 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:32.999+0000 7f6f76eae640 1 --2- 172.21.15.29:0/1351658686 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6f700ff360 0x7f6f7010d200 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f6f6400c860 tx=0x7f6f64004430 comp rx=0 tx=0).stop 2024-03-18T01:18:33.000 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:32.999+0000 7f6f76eae640 1 -- 172.21.15.29:0/1351658686 shutdown_connections 2024-03-18T01:18:33.000 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:32.999+0000 7f6f76eae640 1 --2- 172.21.15.29:0/1351658686 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f6f58069210 0x7f6f5806b6c0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:18:33.001 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:32.999+0000 7f6f76eae640 1 --2- 172.21.15.29:0/1351658686 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6f700ff360 0x7f6f7010d200 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:18:33.001 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:32.999+0000 7f6f76eae640 1 --2- 172.21.15.29:0/1351658686 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6f70065dc0 0x7f6f7010ccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:18:33.001 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:32.999+0000 7f6f76eae640 1 -- 172.21.15.29:0/1351658686 >> 172.21.15.29:0/1351658686 conn(0x7f6f700f6870 msgr2=0x7f6f700f6c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:18:33.001 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:32.999+0000 7f6f76eae640 1 -- 172.21.15.29:0/1351658686 shutdown_connections 2024-03-18T01:18:33.001 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:32.999+0000 7f6f76eae640 1 -- 172.21.15.29:0/1351658686 wait complete. 2024-03-18T01:18:33.011 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-18T01:15:31.263727+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 2, 'pool_name': 'cephfs.cephfs.meta', 'create_time': '2024-03-18T01:18:01.725916+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '44', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 3, 'pool_name': 'cephfs.cephfs.data', 'create_time': '2024-03-18T01:18:02.468256+0000', 'flags': 131073, 'flags_names': 'hashpspool,bulk', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '45', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-18T01:18:33.012 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-18T01:18:33.115 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:33.113+0000 7f8f0c71d640 1 -- 172.21.15.29:0/2707222248 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8f040fc360 msgr2=0x7f8f040fc7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:18:33.115 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:33.113+0000 7f8f0c71d640 1 --2- 172.21.15.29:0/2707222248 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8f040fc360 0x7f8f040fc7d0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f8ef80048d0 tx=0x7f8ef8009da0 comp rx=0 tx=0).stop 2024-03-18T01:18:33.115 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:33.113+0000 7f8f0c71d640 1 -- 172.21.15.29:0/2707222248 shutdown_connections 2024-03-18T01:18:33.115 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:33.113+0000 7f8f0c71d640 1 --2- 172.21.15.29:0/2707222248 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8f040fc360 0x7f8f040fc7d0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:18:33.115 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:33.113+0000 7f8f0c71d640 1 --2- 172.21.15.29:0/2707222248 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8f040fb090 0x7f8f040fb4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:18:33.115 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:33.113+0000 7f8f0c71d640 1 -- 172.21.15.29:0/2707222248 >> 172.21.15.29:0/2707222248 conn(0x7f8f040f6870 msgr2=0x7f8f040f8c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:18:33.115 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:33.113+0000 7f8f0c71d640 1 -- 172.21.15.29:0/2707222248 shutdown_connections 2024-03-18T01:18:33.115 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:33.113+0000 7f8f0c71d640 1 -- 172.21.15.29:0/2707222248 wait complete. 2024-03-18T01:18:33.116 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:33.113+0000 7f8f0c71d640 1 Processor -- start 2024-03-18T01:18:33.116 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:33.113+0000 7f8f0c71d640 1 -- start start 2024-03-18T01:18:33.116 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:33.114+0000 7f8f0c71d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8f040fb090 0x7f8f0410bbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:18:33.116 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:33.114+0000 7f8f0c71d640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8f040fc360 0x7f8f0410c100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:18:33.116 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:33.114+0000 7f8f0c71d640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f0410c750 con 0x7f8f040fb090 2024-03-18T01:18:33.116 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:33.114+0000 7f8f0c71d640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f04111150 con 0x7f8f040fc360 2024-03-18T01:18:33.116 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:33.114+0000 7f8f0a4b9640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8f040fb090 0x7f8f0410bbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:18:33.116 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:33.114+0000 7f8f0a4b9640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8f040fb090 0x7f8f0410bbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:55198/0 (socket says 172.21.15.29:55198) 2024-03-18T01:18:33.116 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:33.114+0000 7f8f0a4b9640 1 -- 172.21.15.29:0/1330414047 learned_addr learned my addr 172.21.15.29:0/1330414047 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-18T01:18:33.116 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:33.114+0000 7f8f09cb8640 1 --2- 172.21.15.29:0/1330414047 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8f040fc360 0x7f8f0410c100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:18:33.116 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:33.114+0000 7f8f0a4b9640 1 -- 172.21.15.29:0/1330414047 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8f040fc360 msgr2=0x7f8f0410c100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:18:33.116 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:33.114+0000 7f8f0a4b9640 1 --2- 172.21.15.29:0/1330414047 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8f040fc360 0x7f8f0410c100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:18:33.116 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:33.114+0000 7f8f0a4b9640 1 -- 172.21.15.29:0/1330414047 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ef8005040 con 0x7f8f040fb090 2024-03-18T01:18:33.116 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:33.114+0000 7f8f0a4b9640 1 --2- 172.21.15.29:0/1330414047 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8f040fb090 0x7f8f0410bbc0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f8efc00c790 tx=0x7f8efc0043c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:18:33.116 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:33.114+0000 7f8f02ffd640 1 -- 172.21.15.29:0/1330414047 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8efc00a7a0 con 0x7f8f040fb090 2024-03-18T01:18:33.117 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:33.114+0000 7f8f02ffd640 1 -- 172.21.15.29:0/1330414047 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8efc00add0 con 0x7f8f040fb090 2024-03-18T01:18:33.117 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:33.114+0000 7f8f0c71d640 1 -- 172.21.15.29:0/1330414047 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f041112f0 con 0x7f8f040fb090 2024-03-18T01:18:33.117 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:33.114+0000 7f8f02ffd640 1 -- 172.21.15.29:0/1330414047 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8efc0115e0 con 0x7f8f040fb090 2024-03-18T01:18:33.117 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:33.114+0000 7f8f0c71d640 1 -- 172.21.15.29:0/1330414047 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f041117e0 con 0x7f8f040fb090 2024-03-18T01:18:33.118 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:33.115+0000 7f8f0c71d640 1 -- 172.21.15.29:0/1330414047 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f04103c00 con 0x7f8f040fb090 2024-03-18T01:18:33.118 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:33.116+0000 7f8f02ffd640 1 -- 172.21.15.29:0/1330414047 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f8efc012400 con 0x7f8f040fb090 2024-03-18T01:18:33.118 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:33.116+0000 7f8f02ffd640 1 --2- 172.21.15.29:0/1330414047 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f8eec069220 0x7f8eec06b6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:18:33.118 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:33.116+0000 7f8f02ffd640 1 -- 172.21.15.29:0/1330414047 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f8efc087290 con 0x7f8f040fb090 2024-03-18T01:18:33.118 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:33.116+0000 7f8f09cb8640 1 --2- 172.21.15.29:0/1330414047 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f8eec069220 0x7f8eec06b6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:18:33.118 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:33.117+0000 7f8f09cb8640 1 --2- 172.21.15.29:0/1330414047 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f8eec069220 0x7f8eec06b6d0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f8ef80048a0 tx=0x7f8ef8009da0 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:18:33.121 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:33.121+0000 7f8f02ffd640 1 -- 172.21.15.29:0/1330414047 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175520 (secure 0 0 0) 0x7f8efc053630 con 0x7f8f040fb090 2024-03-18T01:18:33.290 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:33.290+0000 7f8f0c71d640 1 -- 172.21.15.29:0/1330414047 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f8f04105d20 con 0x7f8f040fb090 2024-03-18T01:18:33.291 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:33.291+0000 7f8f02ffd640 1 -- 172.21.15.29:0/1330414047 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v46) v1 ==== 93+0+10 (secure 0 0 0) 0x7f8efc056c50 con 0x7f8f040fb090 2024-03-18T01:18:33.291 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 1 2024-03-18T01:18:33.293 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:33.292+0000 7f8f0c71d640 1 -- 172.21.15.29:0/1330414047 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f8eec069220 msgr2=0x7f8eec06b6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:18:33.293 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:33.292+0000 7f8f0c71d640 1 --2- 172.21.15.29:0/1330414047 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f8eec069220 0x7f8eec06b6d0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f8ef80048a0 tx=0x7f8ef8009da0 comp rx=0 tx=0).stop 2024-03-18T01:18:33.293 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:33.292+0000 7f8f0c71d640 1 -- 172.21.15.29:0/1330414047 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8f040fb090 msgr2=0x7f8f0410bbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:18:33.293 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:33.292+0000 7f8f0c71d640 1 --2- 172.21.15.29:0/1330414047 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8f040fb090 0x7f8f0410bbc0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f8efc00c790 tx=0x7f8efc0043c0 comp rx=0 tx=0).stop 2024-03-18T01:18:33.293 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:33.292+0000 7f8f0c71d640 1 -- 172.21.15.29:0/1330414047 shutdown_connections 2024-03-18T01:18:33.293 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:33.292+0000 7f8f0c71d640 1 --2- 172.21.15.29:0/1330414047 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f8eec069220 0x7f8eec06b6d0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:18:33.293 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:33.292+0000 7f8f0c71d640 1 --2- 172.21.15.29:0/1330414047 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8f040fc360 0x7f8f0410c100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:18:33.293 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:33.292+0000 7f8f0c71d640 1 --2- 172.21.15.29:0/1330414047 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8f040fb090 0x7f8f0410bbc0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:18:33.293 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:33.292+0000 7f8f0c71d640 1 -- 172.21.15.29:0/1330414047 >> 172.21.15.29:0/1330414047 conn(0x7f8f040f6870 msgr2=0x7f8f040ffaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:18:33.294 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:33.293+0000 7f8f0c71d640 1 -- 172.21.15.29:0/1330414047 shutdown_connections 2024-03-18T01:18:33.294 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:33.293+0000 7f8f0c71d640 1 -- 172.21.15.29:0/1330414047 wait complete. 2024-03-18T01:18:33.303 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.meta pg_num 2024-03-18T01:18:33.406 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:33.404+0000 7f9e1a35d640 1 -- 172.21.15.29:0/3537973991 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9e140fc1a0 msgr2=0x7f9e140fc5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:18:33.406 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:33.404+0000 7f9e1a35d640 1 --2- 172.21.15.29:0/3537973991 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9e140fc1a0 0x7f9e140fc5b0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f9e040048d0 tx=0x7f9e04009da0 comp rx=0 tx=0).stop 2024-03-18T01:18:33.406 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:33.404+0000 7f9e1a35d640 1 -- 172.21.15.29:0/3537973991 shutdown_connections 2024-03-18T01:18:33.406 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:33.404+0000 7f9e1a35d640 1 --2- 172.21.15.29:0/3537973991 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9e140fd470 0x7f9e140fd8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:18:33.406 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:33.404+0000 7f9e1a35d640 1 --2- 172.21.15.29:0/3537973991 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9e140fc1a0 0x7f9e140fc5b0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:18:33.406 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:33.404+0000 7f9e1a35d640 1 -- 172.21.15.29:0/3537973991 >> 172.21.15.29:0/3537973991 conn(0x7f9e140f7960 msgr2=0x7f9e140f9d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:18:33.406 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:33.404+0000 7f9e1a35d640 1 -- 172.21.15.29:0/3537973991 shutdown_connections 2024-03-18T01:18:33.406 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:33.404+0000 7f9e1a35d640 1 -- 172.21.15.29:0/3537973991 wait complete. 2024-03-18T01:18:33.406 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:33.404+0000 7f9e1a35d640 1 Processor -- start 2024-03-18T01:18:33.406 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:33.404+0000 7f9e1a35d640 1 -- start start 2024-03-18T01:18:33.406 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:33.404+0000 7f9e1a35d640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9e140fc1a0 0x7f9e1410ccc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:18:33.407 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:33.404+0000 7f9e1a35d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9e140fd470 0x7f9e1410d200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:18:33.407 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:33.404+0000 7f9e1a35d640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e1410d880 con 0x7f9e140fd470 2024-03-18T01:18:33.407 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:33.404+0000 7f9e1a35d640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e14112240 con 0x7f9e140fc1a0 2024-03-18T01:18:33.407 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:33.405+0000 7f9e137fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9e140fd470 0x7f9e1410d200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:18:33.407 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:33.405+0000 7f9e137fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9e140fd470 0x7f9e1410d200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:55226/0 (socket says 172.21.15.29:55226) 2024-03-18T01:18:33.407 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:33.405+0000 7f9e137fe640 1 -- 172.21.15.29:0/2786397478 learned_addr learned my addr 172.21.15.29:0/2786397478 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-18T01:18:33.407 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:33.405+0000 7f9e13fff640 1 --2- 172.21.15.29:0/2786397478 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9e140fc1a0 0x7f9e1410ccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:18:33.407 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:33.405+0000 7f9e137fe640 1 -- 172.21.15.29:0/2786397478 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9e140fc1a0 msgr2=0x7f9e1410ccc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:18:33.407 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:33.405+0000 7f9e137fe640 1 --2- 172.21.15.29:0/2786397478 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9e140fc1a0 0x7f9e1410ccc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:18:33.407 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:33.405+0000 7f9e137fe640 1 -- 172.21.15.29:0/2786397478 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e04005040 con 0x7f9e140fd470 2024-03-18T01:18:33.407 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:33.405+0000 7f9e137fe640 1 --2- 172.21.15.29:0/2786397478 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9e140fd470 0x7f9e1410d200 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f9e08002a80 tx=0x7f9e08004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:18:33.407 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:33.405+0000 7f9e10ff9640 1 -- 172.21.15.29:0/2786397478 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9e08007c30 con 0x7f9e140fd470 2024-03-18T01:18:33.407 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:33.405+0000 7f9e1a35d640 1 -- 172.21.15.29:0/2786397478 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e141123e0 con 0x7f9e140fd470 2024-03-18T01:18:33.407 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:33.405+0000 7f9e10ff9640 1 -- 172.21.15.29:0/2786397478 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9e0800f040 con 0x7f9e140fd470 2024-03-18T01:18:33.408 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:33.405+0000 7f9e10ff9640 1 -- 172.21.15.29:0/2786397478 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9e0800db80 con 0x7f9e140fd470 2024-03-18T01:18:33.408 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:33.405+0000 7f9e1a35d640 1 -- 172.21.15.29:0/2786397478 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e141128d0 con 0x7f9e140fd470 2024-03-18T01:18:33.408 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:33.406+0000 7f9e1a35d640 1 -- 172.21.15.29:0/2786397478 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e1404bcc0 con 0x7f9e140fd470 2024-03-18T01:18:33.409 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:33.407+0000 7f9e10ff9640 1 -- 172.21.15.29:0/2786397478 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f9e0801c400 con 0x7f9e140fd470 2024-03-18T01:18:33.409 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:33.407+0000 7f9e10ff9640 1 --2- 172.21.15.29:0/2786397478 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f9dfc069210 0x7f9dfc06b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:18:33.409 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:33.407+0000 7f9e10ff9640 1 -- 172.21.15.29:0/2786397478 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f9e08086b10 con 0x7f9e140fd470 2024-03-18T01:18:33.409 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:33.407+0000 7f9e13fff640 1 --2- 172.21.15.29:0/2786397478 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f9dfc069210 0x7f9dfc06b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:18:33.409 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:33.407+0000 7f9e13fff640 1 --2- 172.21.15.29:0/2786397478 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f9dfc069210 0x7f9dfc06b6c0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f9e04000c00 tx=0x7f9e04009da0 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:18:33.411 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:33.411+0000 7f9e10ff9640 1 -- 172.21.15.29:0/2786397478 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175520 (secure 0 0 0) 0x7f9e08052d70 con 0x7f9e140fd470 2024-03-18T01:18:33.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:18:33 smithi029 ceph-mon[29331]: from='client.? 172.21.15.29:0/1351658686' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T01:18:33.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:18:33 smithi029 ceph-mon[29331]: from='client.? 172.21.15.29:0/1330414047' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T01:18:33.583 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:33.583+0000 7f9e1a35d640 1 -- 172.21.15.29:0/2786397478 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"} v 0) v1 -- 0x7f9e14112b20 con 0x7f9e140fd470 2024-03-18T01:18:33.584 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:33.584+0000 7f9e10ff9640 1 -- 172.21.15.29:0/2786397478 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]=0 v46) v1 ==== 107+0+11 (secure 0 0 0) 0x7f9e08056390 con 0x7f9e140fd470 2024-03-18T01:18:33.584 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 32 2024-03-18T01:18:33.586 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:33.586+0000 7f9e1a35d640 1 -- 172.21.15.29:0/2786397478 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f9dfc069210 msgr2=0x7f9dfc06b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:18:33.586 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:33.586+0000 7f9e1a35d640 1 --2- 172.21.15.29:0/2786397478 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f9dfc069210 0x7f9dfc06b6c0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f9e04000c00 tx=0x7f9e04009da0 comp rx=0 tx=0).stop 2024-03-18T01:18:33.586 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:33.586+0000 7f9e1a35d640 1 -- 172.21.15.29:0/2786397478 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9e140fd470 msgr2=0x7f9e1410d200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:18:33.586 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:33.586+0000 7f9e1a35d640 1 --2- 172.21.15.29:0/2786397478 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9e140fd470 0x7f9e1410d200 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f9e08002a80 tx=0x7f9e08004600 comp rx=0 tx=0).stop 2024-03-18T01:18:33.586 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:33.586+0000 7f9e1a35d640 1 -- 172.21.15.29:0/2786397478 shutdown_connections 2024-03-18T01:18:33.586 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:33.586+0000 7f9e1a35d640 1 --2- 172.21.15.29:0/2786397478 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f9dfc069210 0x7f9dfc06b6c0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:18:33.587 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:33.586+0000 7f9e1a35d640 1 --2- 172.21.15.29:0/2786397478 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9e140fd470 0x7f9e1410d200 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:18:33.587 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:33.586+0000 7f9e1a35d640 1 --2- 172.21.15.29:0/2786397478 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9e140fc1a0 0x7f9e1410ccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:18:33.587 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:33.586+0000 7f9e1a35d640 1 -- 172.21.15.29:0/2786397478 >> 172.21.15.29:0/2786397478 conn(0x7f9e140f7960 msgr2=0x7f9e140f9020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:18:33.587 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:33.586+0000 7f9e1a35d640 1 -- 172.21.15.29:0/2786397478 shutdown_connections 2024-03-18T01:18:33.587 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:33.586+0000 7f9e1a35d640 1 -- 172.21.15.29:0/2786397478 wait complete. 2024-03-18T01:18:33.596 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.data pg_num 2024-03-18T01:18:33.699 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:33.697+0000 7f960e35d640 1 -- 172.21.15.29:0/962938519 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f96080faf50 msgr2=0x7f96081017a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:18:33.699 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:33.697+0000 7f960e35d640 1 --2- 172.21.15.29:0/962938519 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f96080faf50 0x7f96081017a0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f95fc0048d0 tx=0x7f95fc009da0 comp rx=0 tx=0).stop 2024-03-18T01:18:33.699 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:33.697+0000 7f960e35d640 1 -- 172.21.15.29:0/962938519 shutdown_connections 2024-03-18T01:18:33.699 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:33.697+0000 7f960e35d640 1 --2- 172.21.15.29:0/962938519 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f96080faf50 0x7f96081017a0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:18:33.699 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:33.697+0000 7f960e35d640 1 --2- 172.21.15.29:0/962938519 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f96080fa600 0x7f96080faa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:18:33.699 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:33.697+0000 7f960e35d640 1 -- 172.21.15.29:0/962938519 >> 172.21.15.29:0/962938519 conn(0x7f96080f6000 msgr2=0x7f96080f8430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:18:33.699 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:33.697+0000 7f960e35d640 1 -- 172.21.15.29:0/962938519 shutdown_connections 2024-03-18T01:18:33.699 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:33.697+0000 7f960e35d640 1 -- 172.21.15.29:0/962938519 wait complete. 2024-03-18T01:18:33.700 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:33.697+0000 7f960e35d640 1 Processor -- start 2024-03-18T01:18:33.700 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:33.698+0000 7f960e35d640 1 -- start start 2024-03-18T01:18:33.700 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:33.698+0000 7f960e35d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f96080fa600 0x7f960810cd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:18:33.700 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:33.698+0000 7f960e35d640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f96080faf50 0x7f960810d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:18:33.700 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:33.698+0000 7f960e35d640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f960810d8c0 con 0x7f96080fa600 2024-03-18T01:18:33.700 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:33.698+0000 7f960e35d640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9608112280 con 0x7f96080faf50 2024-03-18T01:18:33.700 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:33.698+0000 7f9607fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f96080fa600 0x7f960810cd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:18:33.700 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:33.698+0000 7f9607fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f96080fa600 0x7f960810cd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:55242/0 (socket says 172.21.15.29:55242) 2024-03-18T01:18:33.700 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:33.698+0000 7f9607fff640 1 -- 172.21.15.29:0/1654615722 learned_addr learned my addr 172.21.15.29:0/1654615722 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-18T01:18:33.700 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:33.698+0000 7f96077fe640 1 --2- 172.21.15.29:0/1654615722 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f96080faf50 0x7f960810d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:18:33.700 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:33.698+0000 7f9607fff640 1 -- 172.21.15.29:0/1654615722 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f96080faf50 msgr2=0x7f960810d270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:18:33.700 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:33.698+0000 7f9607fff640 1 --2- 172.21.15.29:0/1654615722 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f96080faf50 0x7f960810d270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:18:33.700 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:33.698+0000 7f9607fff640 1 -- 172.21.15.29:0/1654615722 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95fc005040 con 0x7f96080fa600 2024-03-18T01:18:33.700 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:33.698+0000 7f9607fff640 1 --2- 172.21.15.29:0/1654615722 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f96080fa600 0x7f960810cd30 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f95f8002a50 tx=0x7f95f8004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:18:33.701 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:33.698+0000 7f9604ff9640 1 -- 172.21.15.29:0/1654615722 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f95f8007c30 con 0x7f96080fa600 2024-03-18T01:18:33.701 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:33.698+0000 7f960e35d640 1 -- 172.21.15.29:0/1654615722 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9608112420 con 0x7f96080fa600 2024-03-18T01:18:33.701 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:33.698+0000 7f9604ff9640 1 -- 172.21.15.29:0/1654615722 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f95f800f040 con 0x7f96080fa600 2024-03-18T01:18:33.701 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:33.698+0000 7f9604ff9640 1 -- 172.21.15.29:0/1654615722 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f95f800db50 con 0x7f96080fa600 2024-03-18T01:18:33.701 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:33.699+0000 7f960e35d640 1 -- 172.21.15.29:0/1654615722 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96081128f0 con 0x7f96080fa600 2024-03-18T01:18:33.702 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:33.699+0000 7f960e35d640 1 -- 172.21.15.29:0/1654615722 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f960804bcc0 con 0x7f96080fa600 2024-03-18T01:18:33.702 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:33.700+0000 7f9604ff9640 1 -- 172.21.15.29:0/1654615722 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f95f801c400 con 0x7f96080fa600 2024-03-18T01:18:33.702 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:33.700+0000 7f9604ff9640 1 --2- 172.21.15.29:0/1654615722 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f95f0069210 0x7f95f006b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:18:33.702 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:33.700+0000 7f9604ff9640 1 -- 172.21.15.29:0/1654615722 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f95f8086a60 con 0x7f96080fa600 2024-03-18T01:18:33.702 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:33.700+0000 7f96077fe640 1 --2- 172.21.15.29:0/1654615722 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f95f0069210 0x7f95f006b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:18:33.702 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:33.701+0000 7f96077fe640 1 --2- 172.21.15.29:0/1654615722 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f95f0069210 0x7f95f006b6c0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f95fc000c00 tx=0x7f95fc009da0 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:18:33.705 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:33.705+0000 7f9604ff9640 1 -- 172.21.15.29:0/1654615722 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175520 (secure 0 0 0) 0x7f95f804cdb0 con 0x7f96080fa600 2024-03-18T01:18:33.776 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:18:33 smithi090 ceph-mon[36989]: from='client.? 172.21.15.29:0/1351658686' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T01:18:33.776 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:18:33 smithi090 ceph-mon[36989]: from='client.? 172.21.15.29:0/1330414047' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T01:18:33.877 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:33.876+0000 7f960e35d640 1 -- 172.21.15.29:0/1654615722 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"} v 0) v1 -- 0x7f9608112b10 con 0x7f96080fa600 2024-03-18T01:18:33.877 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:33.877+0000 7f9604ff9640 1 -- 172.21.15.29:0/1654615722 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]=0 v46) v1 ==== 107+0+11 (secure 0 0 0) 0x7f95f8052d70 con 0x7f96080fa600 2024-03-18T01:18:33.877 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 32 2024-03-18T01:18:33.879 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:33.879+0000 7f960e35d640 1 -- 172.21.15.29:0/1654615722 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f95f0069210 msgr2=0x7f95f006b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:18:33.879 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:33.879+0000 7f960e35d640 1 --2- 172.21.15.29:0/1654615722 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f95f0069210 0x7f95f006b6c0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f95fc000c00 tx=0x7f95fc009da0 comp rx=0 tx=0).stop 2024-03-18T01:18:33.879 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:33.879+0000 7f960e35d640 1 -- 172.21.15.29:0/1654615722 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f96080fa600 msgr2=0x7f960810cd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:18:33.879 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:33.879+0000 7f960e35d640 1 --2- 172.21.15.29:0/1654615722 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f96080fa600 0x7f960810cd30 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f95f8002a50 tx=0x7f95f8004600 comp rx=0 tx=0).stop 2024-03-18T01:18:33.879 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:33.879+0000 7f960e35d640 1 -- 172.21.15.29:0/1654615722 shutdown_connections 2024-03-18T01:18:33.879 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:33.879+0000 7f960e35d640 1 --2- 172.21.15.29:0/1654615722 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f95f0069210 0x7f95f006b6c0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:18:33.880 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:33.879+0000 7f960e35d640 1 --2- 172.21.15.29:0/1654615722 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f96080faf50 0x7f960810d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:18:33.880 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:33.879+0000 7f960e35d640 1 --2- 172.21.15.29:0/1654615722 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f96080fa600 0x7f960810cd30 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:18:33.880 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:33.879+0000 7f960e35d640 1 -- 172.21.15.29:0/1654615722 >> 172.21.15.29:0/1654615722 conn(0x7f96080f6000 msgr2=0x7f96080ff860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:18:33.880 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:33.879+0000 7f960e35d640 1 -- 172.21.15.29:0/1654615722 shutdown_connections 2024-03-18T01:18:33.880 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:33.879+0000 7f960e35d640 1 -- 172.21.15.29:0/1654615722 wait complete. 2024-03-18T01:18:33.890 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T01:18:33.989 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:33.987+0000 7fa845d9d640 1 -- 172.21.15.29:0/3987500389 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa8400fd880 msgr2=0x7fa8400fdc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:18:33.990 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:33.987+0000 7fa845d9d640 1 --2- 172.21.15.29:0/3987500389 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa8400fd880 0x7fa8400fdc90 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fa8300048d0 tx=0x7fa830009da0 comp rx=0 tx=0).stop 2024-03-18T01:18:33.990 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:33.987+0000 7fa845d9d640 1 -- 172.21.15.29:0/3987500389 shutdown_connections 2024-03-18T01:18:33.990 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:33.987+0000 7fa845d9d640 1 --2- 172.21.15.29:0/3987500389 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa8400fe1d0 0x7fa8400fa330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:18:33.990 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:33.987+0000 7fa845d9d640 1 --2- 172.21.15.29:0/3987500389 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa8400fd880 0x7fa8400fdc90 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:18:33.990 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:33.987+0000 7fa845d9d640 1 -- 172.21.15.29:0/3987500389 >> 172.21.15.29:0/3987500389 conn(0x7fa8400f6080 msgr2=0x7fa8400f84b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:18:33.990 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:33.987+0000 7fa845d9d640 1 -- 172.21.15.29:0/3987500389 shutdown_connections 2024-03-18T01:18:33.990 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:33.987+0000 7fa845d9d640 1 -- 172.21.15.29:0/3987500389 wait complete. 2024-03-18T01:18:33.990 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:33.988+0000 7fa845d9d640 1 Processor -- start 2024-03-18T01:18:33.990 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:33.988+0000 7fa845d9d640 1 -- start start 2024-03-18T01:18:33.990 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:33.988+0000 7fa845d9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa8400fd880 0x7fa84010cdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:18:33.990 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:33.988+0000 7fa845d9d640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa8400fe1d0 0x7fa84010d2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:18:33.990 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:33.988+0000 7fa845d9d640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa84010d940 con 0x7fa8400fd880 2024-03-18T01:18:33.991 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:33.988+0000 7fa845d9d640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa840112300 con 0x7fa8400fe1d0 2024-03-18T01:18:33.991 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:33.988+0000 7fa83f7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa8400fd880 0x7fa84010cdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:18:33.991 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:33.988+0000 7fa83f7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa8400fd880 0x7fa84010cdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:55264/0 (socket says 172.21.15.29:55264) 2024-03-18T01:18:33.991 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:33.988+0000 7fa83f7fe640 1 -- 172.21.15.29:0/1871546574 learned_addr learned my addr 172.21.15.29:0/1871546574 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-18T01:18:33.991 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:33.988+0000 7fa83effd640 1 --2- 172.21.15.29:0/1871546574 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa8400fe1d0 0x7fa84010d2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:18:33.991 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:33.989+0000 7fa83f7fe640 1 -- 172.21.15.29:0/1871546574 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa8400fe1d0 msgr2=0x7fa84010d2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:18:33.991 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:33.989+0000 7fa83f7fe640 1 --2- 172.21.15.29:0/1871546574 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa8400fe1d0 0x7fa84010d2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:18:33.991 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:33.989+0000 7fa83f7fe640 1 -- 172.21.15.29:0/1871546574 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa830005040 con 0x7fa8400fd880 2024-03-18T01:18:33.991 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:33.989+0000 7fa83effd640 1 --2- 172.21.15.29:0/1871546574 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa8400fe1d0 0x7fa84010d2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T01:18:33.991 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:33.989+0000 7fa83f7fe640 1 --2- 172.21.15.29:0/1871546574 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa8400fd880 0x7fa84010cdb0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fa8300048a0 tx=0x7fa830009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:18:33.991 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:33.989+0000 7fa827fff640 1 -- 172.21.15.29:0/1871546574 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa8300092e0 con 0x7fa8400fd880 2024-03-18T01:18:33.991 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:33.989+0000 7fa827fff640 1 -- 172.21.15.29:0/1871546574 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa83003d450 con 0x7fa8400fd880 2024-03-18T01:18:33.991 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:33.989+0000 7fa845d9d640 1 -- 172.21.15.29:0/1871546574 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa840112440 con 0x7fa8400fd880 2024-03-18T01:18:33.991 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:33.989+0000 7fa827fff640 1 -- 172.21.15.29:0/1871546574 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa83003b640 con 0x7fa8400fd880 2024-03-18T01:18:33.991 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:33.989+0000 7fa845d9d640 1 -- 172.21.15.29:0/1871546574 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa840112910 con 0x7fa8400fd880 2024-03-18T01:18:33.992 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:33.990+0000 7fa845d9d640 1 -- 172.21.15.29:0/1871546574 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa84004bcc0 con 0x7fa8400fd880 2024-03-18T01:18:33.992 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:33.990+0000 7fa827fff640 1 -- 172.21.15.29:0/1871546574 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7fa83002f070 con 0x7fa8400fd880 2024-03-18T01:18:33.992 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:33.991+0000 7fa827fff640 1 --2- 172.21.15.29:0/1871546574 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7fa828069210 0x7fa82806b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:18:33.993 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:33.991+0000 7fa83effd640 1 --2- 172.21.15.29:0/1871546574 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7fa828069210 0x7fa82806b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:18:33.993 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:33.991+0000 7fa827fff640 1 -- 172.21.15.29:0/1871546574 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7fa8300a9630 con 0x7fa8400fd880 2024-03-18T01:18:33.993 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:33.991+0000 7fa83effd640 1 --2- 172.21.15.29:0/1871546574 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7fa828069210 0x7fa82806b6c0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fa8400654c0 tx=0x7fa83400b040 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:18:33.995 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:33.995+0000 7fa827fff640 1 -- 172.21.15.29:0/1871546574 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175520 (secure 0 0 0) 0x7fa830075910 con 0x7fa8400fd880 2024-03-18T01:18:34.206 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:34.205+0000 7fa845d9d640 1 -- 172.21.15.29:0/1871546574 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa840112af0 con 0x7fa8400fd880 2024-03-18T01:18:34.206 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:34.206+0000 7fa827fff640 1 -- 172.21.15.29:0/1871546574 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) v1 ==== 94+0+4713 (secure 0 0 0) 0x7fa830078f30 con 0x7fa8400fd880 2024-03-18T01:18:34.207 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-18T01:18:34.207 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":20,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-18T01:18:03.228199+0000","modified":"2024-03-18T01:18:21.293804+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14480,"mds_1":24281},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14480":{"gid":14480,"name":"cephfs.smithi029.brxqeh","rank":0,"incarnation":15,"state":"up:active","state_seq":6,"addr":"172.21.15.29:6829/1299751620","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":1299751620},{"type":"v1","addr":"172.21.15.29:6829","nonce":1299751620}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_14492":{"gid":14492,"name":"cephfs.smithi029.xuxvlr","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6827/890079928","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6826","nonce":890079928},{"type":"v1","addr":"172.21.15.29:6827","nonce":890079928}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24281":{"gid":24281,"name":"cephfs.smithi090.bohder","rank":1,"incarnation":8,"state":"up:active","state_seq":4,"addr":"172.21.15.90:6825/1998449484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6824","nonce":1998449484},{"type":"v1","addr":"172.21.15.90:6825","nonce":1998449484}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24297":{"gid":24297,"name":"cephfs.smithi090.trsnxz","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":2,"addr":"172.21.15.90:6827/439410941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":439410941},{"type":"v1","addr":"172.21.15.90:6827","nonce":439410941}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2024-03-18T01:18:34.207 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 20 2024-03-18T01:18:34.208 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:34.208+0000 7fa845d9d640 1 -- 172.21.15.29:0/1871546574 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7fa828069210 msgr2=0x7fa82806b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:18:34.208 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:34.208+0000 7fa845d9d640 1 --2- 172.21.15.29:0/1871546574 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7fa828069210 0x7fa82806b6c0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fa8400654c0 tx=0x7fa83400b040 comp rx=0 tx=0).stop 2024-03-18T01:18:34.209 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:34.208+0000 7fa845d9d640 1 -- 172.21.15.29:0/1871546574 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa8400fd880 msgr2=0x7fa84010cdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:18:34.209 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:34.208+0000 7fa845d9d640 1 --2- 172.21.15.29:0/1871546574 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa8400fd880 0x7fa84010cdb0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fa8300048a0 tx=0x7fa830009da0 comp rx=0 tx=0).stop 2024-03-18T01:18:34.209 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:34.208+0000 7fa845d9d640 1 -- 172.21.15.29:0/1871546574 shutdown_connections 2024-03-18T01:18:34.209 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:34.208+0000 7fa845d9d640 1 --2- 172.21.15.29:0/1871546574 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7fa828069210 0x7fa82806b6c0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:18:34.209 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:34.208+0000 7fa845d9d640 1 --2- 172.21.15.29:0/1871546574 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa8400fe1d0 0x7fa84010d2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:18:34.209 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:34.208+0000 7fa845d9d640 1 --2- 172.21.15.29:0/1871546574 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa8400fd880 0x7fa84010cdb0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:18:34.209 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:34.208+0000 7fa845d9d640 1 -- 172.21.15.29:0/1871546574 >> 172.21.15.29:0/1871546574 conn(0x7fa8400f6080 msgr2=0x7fa8400f77d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:18:34.209 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:34.208+0000 7fa845d9d640 1 -- 172.21.15.29:0/1871546574 shutdown_connections 2024-03-18T01:18:34.209 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:34.208+0000 7fa845d9d640 1 -- 172.21.15.29:0/1871546574 wait complete. 2024-03-18T01:18:34.219 DEBUG:tasks.fs:fs fscid=1,name=cephfs state = {'epoch': 20, 'max_mds': 2, 'flags': 50} 2024-03-18T01:18:34.219 INFO:teuthology.run_tasks:Running task kclient... 2024-03-18T01:18:34.243 INFO:tasks.kclient:Mounting kernel clients... 2024-03-18T01:18:34.244 INFO:tasks.kclient:config is {'client.0': {}, 'client.1': {}} 2024-03-18T01:18:34.244 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T01:18:34.244 DEBUG:teuthology.orchestra.run.smithi029:> ip netns list 2024-03-18T01:18:34.259 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T01:18:34.259 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip link delete ceph-brx 2024-03-18T01:18:34.331 INFO:teuthology.orchestra.run.smithi029.stderr:Cannot find device "ceph-brx" 2024-03-18T01:18:34.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-18T01:18:34.333 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T01:18:34.333 DEBUG:teuthology.orchestra.run.smithi090:> ip netns list 2024-03-18T01:18:34.353 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T01:18:34.353 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip link delete ceph-brx 2024-03-18T01:18:34.435 INFO:teuthology.orchestra.run.smithi090.stderr:Cannot find device "ceph-brx" 2024-03-18T01:18:34.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-18T01:18:34.437 INFO:tasks.kclient:client.0 config is {'syntax': 'v1'} 2024-03-18T01:18:34.437 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-03-18T01:18:34.437 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-18T01:18:34.437 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 37f51182-e4c4-11ee-95c9-87774f69a715 -- ceph fs ls 2024-03-18T01:18:34.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:18:34 smithi029 ceph-mon[29331]: pgmap v195: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 6.1 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-18T01:18:34.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:18:34 smithi029 ceph-mon[29331]: from='client.? 172.21.15.29:0/2786397478' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2024-03-18T01:18:34.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:18:34 smithi029 ceph-mon[29331]: from='client.? 172.21.15.29:0/1654615722' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2024-03-18T01:18:34.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:18:34 smithi029 ceph-mon[29331]: from='client.? 172.21.15.29:0/1871546574' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T01:18:34.698 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/37f51182-e4c4-11ee-95c9-87774f69a715/mon.smithi029/config 2024-03-18T01:18:34.776 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:18:34 smithi090 ceph-mon[36989]: pgmap v195: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 6.1 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-18T01:18:34.776 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:18:34 smithi090 ceph-mon[36989]: from='client.? 172.21.15.29:0/2786397478' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2024-03-18T01:18:34.777 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:18:34 smithi090 ceph-mon[36989]: from='client.? 172.21.15.29:0/1654615722' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2024-03-18T01:18:34.777 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:18:34 smithi090 ceph-mon[36989]: from='client.? 172.21.15.29:0/1871546574' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T01:18:36.334 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:36.332+0000 7fbc2c0b1700 1 -- 172.21.15.29:0/2694325730 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbc241037a0 msgr2=0x7fbc240fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:18:36.335 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:36.332+0000 7fbc2c0b1700 1 --2- 172.21.15.29:0/2694325730 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbc241037a0 0x7fbc240fce10 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fbc14004940 tx=0x7fbc14009e30 comp rx=0 tx=0).stop 2024-03-18T01:18:36.335 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:36.332+0000 7fbc2c0b1700 1 -- 172.21.15.29:0/2694325730 shutdown_connections 2024-03-18T01:18:36.335 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:36.332+0000 7fbc2c0b1700 1 --2- 172.21.15.29:0/2694325730 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbc241037a0 0x7fbc240fce10 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:18:36.335 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:36.332+0000 7fbc2c0b1700 1 --2- 172.21.15.29:0/2694325730 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbc24102e00 0x7fbc241031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:18:36.335 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:36.332+0000 7fbc2c0b1700 1 -- 172.21.15.29:0/2694325730 >> 172.21.15.29:0/2694325730 conn(0x7fbc240f89b0 msgr2=0x7fbc240fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:18:36.335 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:36.333+0000 7fbc2c0b1700 1 -- 172.21.15.29:0/2694325730 shutdown_connections 2024-03-18T01:18:36.335 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:36.333+0000 7fbc2c0b1700 1 -- 172.21.15.29:0/2694325730 wait complete. 2024-03-18T01:18:36.335 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:36.333+0000 7fbc2c0b1700 1 Processor -- start 2024-03-18T01:18:36.335 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:36.333+0000 7fbc2c0b1700 1 -- start start 2024-03-18T01:18:36.335 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:36.333+0000 7fbc2c0b1700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbc24102e00 0x7fbc240702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:18:36.335 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:36.333+0000 7fbc2c0b1700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbc241037a0 0x7fbc24070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:18:36.336 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:36.333+0000 7fbc2c0b1700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc24072210 con 0x7fbc24102e00 2024-03-18T01:18:36.336 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:36.333+0000 7fbc2c0b1700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc24070d40 con 0x7fbc241037a0 2024-03-18T01:18:36.336 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:36.333+0000 7fbc29e4d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbc24102e00 0x7fbc240702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:18:36.336 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:36.333+0000 7fbc29e4d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbc24102e00 0x7fbc240702c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:55276/0 (socket says 172.21.15.29:55276) 2024-03-18T01:18:36.336 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:36.333+0000 7fbc29e4d700 1 -- 172.21.15.29:0/7214598 learned_addr learned my addr 172.21.15.29:0/7214598 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-18T01:18:36.336 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:36.333+0000 7fbc2964c700 1 --2- 172.21.15.29:0/7214598 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbc241037a0 0x7fbc24070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:18:36.336 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:36.334+0000 7fbc29e4d700 1 -- 172.21.15.29:0/7214598 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbc241037a0 msgr2=0x7fbc24070800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:18:36.336 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:36.334+0000 7fbc29e4d700 1 --2- 172.21.15.29:0/7214598 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbc241037a0 0x7fbc24070800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:18:36.336 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:36.334+0000 7fbc29e4d700 1 -- 172.21.15.29:0/7214598 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc14005040 con 0x7fbc24102e00 2024-03-18T01:18:36.336 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:36.334+0000 7fbc29e4d700 1 --2- 172.21.15.29:0/7214598 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbc24102e00 0x7fbc240702c0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fbc20002a50 tx=0x7fbc200046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:18:36.336 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:36.334+0000 7fbc1a7fc700 1 -- 172.21.15.29:0/7214598 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbc20007cc0 con 0x7fbc24102e00 2024-03-18T01:18:36.336 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:36.334+0000 7fbc1a7fc700 1 -- 172.21.15.29:0/7214598 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fbc2000f040 con 0x7fbc24102e00 2024-03-18T01:18:36.336 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:36.334+0000 7fbc2c0b1700 1 -- 172.21.15.29:0/7214598 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc24071020 con 0x7fbc24102e00 2024-03-18T01:18:36.337 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:36.334+0000 7fbc1a7fc700 1 -- 172.21.15.29:0/7214598 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbc2000da30 con 0x7fbc24102e00 2024-03-18T01:18:36.337 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:36.334+0000 7fbc2c0b1700 1 -- 172.21.15.29:0/7214598 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc2411a130 con 0x7fbc24102e00 2024-03-18T01:18:36.337 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:36.335+0000 7fbc2c0b1700 1 -- 172.21.15.29:0/7214598 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc241052d0 con 0x7fbc24102e00 2024-03-18T01:18:36.338 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:36.336+0000 7fbc1a7fc700 1 -- 172.21.15.29:0/7214598 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7fbc2000db90 con 0x7fbc24102e00 2024-03-18T01:18:36.338 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:36.336+0000 7fbc1a7fc700 1 --2- 172.21.15.29:0/7214598 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7fbc100692a0 0x7fbc1006b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:18:36.338 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:36.336+0000 7fbc2964c700 1 --2- 172.21.15.29:0/7214598 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7fbc100692a0 0x7fbc1006b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:18:36.338 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:36.336+0000 7fbc1a7fc700 1 -- 172.21.15.29:0/7214598 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7fbc20086eb0 con 0x7fbc24102e00 2024-03-18T01:18:36.338 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:36.336+0000 7fbc2964c700 1 --2- 172.21.15.29:0/7214598 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7fbc100692a0 0x7fbc1006b750 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fbc240719f0 tx=0x7fbc14009e30 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:18:36.340 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:36.340+0000 7fbc1a7fc700 1 -- 172.21.15.29:0/7214598 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175520 (secure 0 0 0) 0x7fbc20053190 con 0x7fbc24102e00 2024-03-18T01:18:36.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:18:36 smithi029 ceph-mon[29331]: pgmap v196: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 5.5 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-18T01:18:36.579 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:36.578+0000 7fbc2c0b1700 1 -- 172.21.15.29:0/7214598 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fbc240717a0 con 0x7fbc24102e00 2024-03-18T01:18:36.579 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:36.579+0000 7fbc1a7fc700 1 -- 172.21.15.29:0/7214598 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v20) v1 ==== 53+0+83 (secure 0 0 0) 0x7fbc200567b0 con 0x7fbc24102e00 2024-03-18T01:18:36.579 INFO:teuthology.orchestra.run.smithi029.stdout:name: cephfs, metadata pool: cephfs.cephfs.meta, data pools: [cephfs.cephfs.data ] 2024-03-18T01:18:36.581 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:36.581+0000 7fbc2c0b1700 1 -- 172.21.15.29:0/7214598 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7fbc100692a0 msgr2=0x7fbc1006b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:18:36.581 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:36.581+0000 7fbc2c0b1700 1 --2- 172.21.15.29:0/7214598 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7fbc100692a0 0x7fbc1006b750 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fbc240719f0 tx=0x7fbc14009e30 comp rx=0 tx=0).stop 2024-03-18T01:18:36.581 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:36.581+0000 7fbc2c0b1700 1 -- 172.21.15.29:0/7214598 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbc24102e00 msgr2=0x7fbc240702c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:18:36.581 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:36.581+0000 7fbc2c0b1700 1 --2- 172.21.15.29:0/7214598 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbc24102e00 0x7fbc240702c0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fbc20002a50 tx=0x7fbc200046b0 comp rx=0 tx=0).stop 2024-03-18T01:18:36.581 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:36.581+0000 7fbc2c0b1700 1 -- 172.21.15.29:0/7214598 shutdown_connections 2024-03-18T01:18:36.582 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:36.581+0000 7fbc2c0b1700 1 --2- 172.21.15.29:0/7214598 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7fbc100692a0 0x7fbc1006b750 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:18:36.582 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:36.581+0000 7fbc2c0b1700 1 --2- 172.21.15.29:0/7214598 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbc24102e00 0x7fbc240702c0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:18:36.582 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:36.581+0000 7fbc2c0b1700 1 --2- 172.21.15.29:0/7214598 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbc241037a0 0x7fbc24070800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:18:36.582 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:36.581+0000 7fbc2c0b1700 1 -- 172.21.15.29:0/7214598 >> 172.21.15.29:0/7214598 conn(0x7fbc240f89b0 msgr2=0x7fbc241021d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:18:36.582 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:36.581+0000 7fbc2c0b1700 1 -- 172.21.15.29:0/7214598 shutdown_connections 2024-03-18T01:18:36.582 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:36.581+0000 7fbc2c0b1700 1 -- 172.21.15.29:0/7214598 wait complete. 2024-03-18T01:18:36.776 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:18:36 smithi090 ceph-mon[36989]: pgmap v196: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 5.5 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-18T01:18:37.274 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-18T01:18:37.274 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-18T01:18:37.274 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi029.front.sepia.ceph.com 2024-03-18T01:18:37.274 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-18T01:18:37.274 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-18T01:18:37.274 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-18T01:18:37.274 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-18T01:18:37.274 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-18T01:18:37.274 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-18T01:18:37.274 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T01:18:37.275 DEBUG:teuthology.orchestra.run.smithi029:> ip addr 2024-03-18T01:18:37.291 INFO:teuthology.orchestra.run.smithi029.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-18T01:18:37.292 INFO:teuthology.orchestra.run.smithi029.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-18T01:18:37.292 INFO:teuthology.orchestra.run.smithi029.stdout: inet 127.0.0.1/8 scope host lo 2024-03-18T01:18:37.292 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2024-03-18T01:18:37.292 INFO:teuthology.orchestra.run.smithi029.stdout: inet6 ::1/128 scope host 2024-03-18T01:18:37.292 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2024-03-18T01:18:37.292 INFO:teuthology.orchestra.run.smithi029.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-18T01:18:37.292 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 0c:c4:7a:6b:fd:a0 brd ff:ff:ff:ff:ff:ff 2024-03-18T01:18:37.292 INFO:teuthology.orchestra.run.smithi029.stdout: altname enp7s0f0 2024-03-18T01:18:37.293 INFO:teuthology.orchestra.run.smithi029.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-18T01:18:37.293 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 0c:c4:7a:6b:fd:a1 brd ff:ff:ff:ff:ff:ff 2024-03-18T01:18:37.293 INFO:teuthology.orchestra.run.smithi029.stdout: altname enp7s0f1 2024-03-18T01:18:37.293 INFO:teuthology.orchestra.run.smithi029.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-18T01:18:37.293 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 0c:c4:7a:bd:12:5a brd ff:ff:ff:ff:ff:ff 2024-03-18T01:18:37.293 INFO:teuthology.orchestra.run.smithi029.stdout: altname enp1s0f0 2024-03-18T01:18:37.293 INFO:teuthology.orchestra.run.smithi029.stdout: inet 172.21.15.29/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-03-18T01:18:37.293 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft 42207sec preferred_lft 42207sec 2024-03-18T01:18:37.293 INFO:teuthology.orchestra.run.smithi029.stdout: inet6 fe80::ec4:7aff:febd:125a/64 scope link noprefixroute 2024-03-18T01:18:37.293 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2024-03-18T01:18:37.293 INFO:teuthology.orchestra.run.smithi029.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-18T01:18:37.293 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 0c:c4:7a:bd:12:5b brd ff:ff:ff:ff:ff:ff 2024-03-18T01:18:37.294 INFO:teuthology.orchestra.run.smithi029.stdout: altname enp1s0f1 2024-03-18T01:18:37.294 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-03-18T01:18:37.294 DEBUG:teuthology.orchestra.run.smithi029:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-18T01:18:37.294 DEBUG:teuthology.orchestra.run.smithi029:> set -e 2024-03-18T01:18:37.294 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip link add name ceph-brx type bridge 2024-03-18T01:18:37.295 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip addr flush dev ceph-brx 2024-03-18T01:18:37.295 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip link set ceph-brx up 2024-03-18T01:18:37.295 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-03-18T01:18:37.295 DEBUG:teuthology.orchestra.run.smithi029:> ') 2024-03-18T01:18:37.383 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:37 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-18T01:18:37.498 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:18:37 smithi029 ceph-mon[29331]: from='client.? 172.21.15.29:0/7214598' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-18T01:18:37.501 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:37 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-18T01:18:37.507 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T01:18:37.508 DEBUG:teuthology.orchestra.run.smithi029:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-18T01:18:37.534 INFO:teuthology.orchestra.run.smithi029.stdout:1 2024-03-18T01:18:37.536 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T01:18:37.536 DEBUG:teuthology.orchestra.run.smithi029:> route 2024-03-18T01:18:37.610 INFO:teuthology.orchestra.run.smithi029.stdout:Kernel IP routing table 2024-03-18T01:18:37.610 INFO:teuthology.orchestra.run.smithi029.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-18T01:18:37.611 INFO:teuthology.orchestra.run.smithi029.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-03-18T01:18:37.611 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-03-18T01:18:37.611 INFO:teuthology.orchestra.run.smithi029.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-18T01:18:37.611 DEBUG:teuthology.orchestra.run.smithi029:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-18T01:18:37.612 DEBUG:teuthology.orchestra.run.smithi029:> set -e 2024-03-18T01:18:37.612 DEBUG:teuthology.orchestra.run.smithi029:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-03-18T01:18:37.612 DEBUG:teuthology.orchestra.run.smithi029:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-03-18T01:18:37.613 DEBUG:teuthology.orchestra.run.smithi029:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-03-18T01:18:37.613 DEBUG:teuthology.orchestra.run.smithi029:> ') 2024-03-18T01:18:37.698 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:37 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-18T01:18:37.776 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:18:37 smithi090 ceph-mon[36989]: from='client.? 172.21.15.29:0/7214598' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-18T01:18:37.914 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:37 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-18T01:18:37.920 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T01:18:37.920 DEBUG:teuthology.orchestra.run.smithi029:> ip netns list 2024-03-18T01:18:37.934 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T01:18:37.935 DEBUG:teuthology.orchestra.run.smithi029:> ip netns list-id 2024-03-18T01:18:37.989 DEBUG:teuthology.orchestra.run.smithi029:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-18T01:18:37.989 DEBUG:teuthology.orchestra.run.smithi029:> set -e 2024-03-18T01:18:37.989 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-18T01:18:37.989 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-03-18T01:18:37.989 DEBUG:teuthology.orchestra.run.smithi029:> ') 2024-03-18T01:18:38.076 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:38 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-18T01:18:38.105 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:38 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-18T01:18:38.111 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-03-18T01:18:38.111 DEBUG:teuthology.orchestra.run.smithi029:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-18T01:18:38.111 DEBUG:teuthology.orchestra.run.smithi029:> set -e 2024-03-18T01:18:38.111 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-03-18T01:18:38.112 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-03-18T01:18:38.112 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-03-18T01:18:38.112 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-03-18T01:18:38.112 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-03-18T01:18:38.112 DEBUG:teuthology.orchestra.run.smithi029:> ') 2024-03-18T01:18:38.160 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:38 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-18T01:18:38.247 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:38 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-18T01:18:38.253 DEBUG:teuthology.orchestra.run.smithi029:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-18T01:18:38.253 DEBUG:teuthology.orchestra.run.smithi029:> set -e 2024-03-18T01:18:38.253 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip link set brx.0 up 2024-03-18T01:18:38.254 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip link set dev brx.0 master ceph-brx 2024-03-18T01:18:38.254 DEBUG:teuthology.orchestra.run.smithi029:> ') 2024-03-18T01:18:38.301 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:38 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-18T01:18:38.328 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:38 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-18T01:18:38.335 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-18T01:18:38.335 DEBUG:teuthology.orchestra.run.smithi029:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-18T01:18:38.349 INFO:teuthology.orchestra.run.smithi029.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-18T01:18:38.349 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-18T01:18:38.350 DEBUG:teuthology.orchestra.run.smithi029:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-18T01:18:38.403 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-03-18T01:18:38.403 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T01:18:38.403 DEBUG:teuthology.orchestra.run.smithi029:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-03-18T01:18:38.466 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:18:38 smithi029 ceph-mon[29331]: pgmap v197: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.9 KiB/s rd, 0 B/s wr, 5 op/s 2024-03-18T01:18:38.496 INFO:teuthology.orchestra.run.smithi029.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-03-18T01:18:38.496 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-03-18T01:18:38.496 INFO:teuthology.orchestra.run.smithi029.stdout:invalid new device string format 2024-03-18T01:18:38.598 INFO:teuthology.orchestra.run.smithi029.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-03-18T01:18:38.598 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-03-18T01:18:38.599 INFO:teuthology.orchestra.run.smithi029.stdout:invalid new device string format 2024-03-18T01:18:38.599 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: resolved to: "172.21.15.29:3300,172.21.15.90:3300" 2024-03-18T01:18:38.599 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: switching to using v1 address with old syntax 2024-03-18T01:18:38.604 INFO:teuthology.orchestra.run.smithi029.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-03-18T01:18:38.604 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-03-18T01:18:38.604 INFO:teuthology.orchestra.run.smithi029.stdout:invalid new device string format 2024-03-18T01:18:38.604 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: resolved to: "172.21.15.29:3300,172.21.15.90:3300" 2024-03-18T01:18:38.604 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: switching to using v1 address with old syntax 2024-03-18T01:18:38.604 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: resolved to: "172.21.15.29:6789,172.21.15.90:6789" 2024-03-18T01:18:38.604 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.29:6789,172.21.15.90:6789:/ 2024-03-18T01:18:38.604 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=37f51182-e4c4-11ee-95c9-87774f69a715" will pass to kernel 2024-03-18T01:18:38.605 INFO:teuthology.orchestra.run.smithi029.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-03-18T01:18:38.605 INFO:teuthology.orchestra.run.smithi029.stdout: You just mounted a file system that supports labels which does not 2024-03-18T01:18:38.605 INFO:teuthology.orchestra.run.smithi029.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-03-18T01:18:38.605 INFO:teuthology.orchestra.run.smithi029.stdout: applications will generate AVC messages and not be allowed access to 2024-03-18T01:18:38.605 INFO:teuthology.orchestra.run.smithi029.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-03-18T01:18:38.607 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-03-18T01:18:38.607 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T01:18:38.607 DEBUG:teuthology.orchestra.run.smithi029:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-18T01:18:38.675 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-18T01:18:38.677 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:18:38 smithi090 ceph-mon[36989]: pgmap v197: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.9 KiB/s rd, 0 B/s wr, 5 op/s 2024-03-18T01:18:38.765 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:38 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-03-18T01:18:38.770 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:38 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-18T01:18:38.776 INFO:teuthology.orchestra.run.smithi029.stdout:client24329 2024-03-18T01:18:38.777 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-03-18T01:18:38.777 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-18T01:18:38.793 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-18T01:18:38.882 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:38 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-03-18T01:18:38.887 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:38 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-18T01:18:38.893 INFO:teuthology.orchestra.run.smithi029.stdout:client24329 2024-03-18T01:18:38.894 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-18T01:18:38.943 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:38 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-03-18T01:18:38.948 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:38 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-18T01:18:38.954 INFO:teuthology.orchestra.run.smithi029.stdout:client24329 2024-03-18T01:18:38.954 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=300 -- bash -c 'sudo dd if=/sys/kernel/debug/ceph/37f51182-e4c4-11ee-95c9-87774f69a715.client24329/status') 2024-03-18T01:18:39.003 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:39 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-18T01:18:39.018 INFO:teuthology.orchestra.run.smithi029.stdout:instance: client.24329 (3)172.21.15.29:0/3008187387 2024-03-18T01:18:39.018 INFO:teuthology.orchestra.run.smithi029.stdout:blocklisted: false 2024-03-18T01:18:39.018 INFO:teuthology.orchestra.run.smithi029.stderr:0+1 records in 2024-03-18T01:18:39.018 INFO:teuthology.orchestra.run.smithi029.stderr:0+1 records out 2024-03-18T01:18:39.019 INFO:teuthology.orchestra.run.smithi029.stderr:71 bytes copied, 2.0528e-05 s, 3.5 MB/s 2024-03-18T01:18:39.019 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:39 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-18T01:18:39.029 INFO:tasks.kclient:client.1 config is {'syntax': 'v1'} 2024-03-18T01:18:39.030 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-03-18T01:18:39.030 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-03-18T01:18:39.030 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 37f51182-e4c4-11ee-95c9-87774f69a715 -- ceph fs ls 2024-03-18T01:18:39.295 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/37f51182-e4c4-11ee-95c9-87774f69a715/mon.smithi029/config 2024-03-18T01:18:40.754 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:18:40 smithi029 ceph-mon[29331]: pgmap v198: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-18T01:18:40.776 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:18:40 smithi090 ceph-mon[36989]: pgmap v198: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-18T01:18:41.105 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:41.099+0000 7fdd42158700 1 -- 172.21.15.29:0/2958811837 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdd3c102f50 msgr2=0x7fdd3c103320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:18:41.105 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:41.099+0000 7fdd42158700 1 --2- 172.21.15.29:0/2958811837 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdd3c102f50 0x7fdd3c103320 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fdd30004940 tx=0x7fdd30009e30 comp rx=0 tx=0).stop 2024-03-18T01:18:41.105 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:41.099+0000 7fdd42158700 1 -- 172.21.15.29:0/2958811837 shutdown_connections 2024-03-18T01:18:41.105 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:41.099+0000 7fdd42158700 1 --2- 172.21.15.29:0/2958811837 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdd3c0fb4d0 0x7fdd3c0fb940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:18:41.105 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:41.099+0000 7fdd42158700 1 --2- 172.21.15.29:0/2958811837 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdd3c102f50 0x7fdd3c103320 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:18:41.106 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:41.099+0000 7fdd42158700 1 -- 172.21.15.29:0/2958811837 >> 172.21.15.29:0/2958811837 conn(0x7fdd3c0714c0 msgr2=0x7fdd3c0718c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:18:41.106 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:41.099+0000 7fdd42158700 1 -- 172.21.15.29:0/2958811837 shutdown_connections 2024-03-18T01:18:41.106 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:41.099+0000 7fdd42158700 1 -- 172.21.15.29:0/2958811837 wait complete. 2024-03-18T01:18:41.106 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:41.100+0000 7fdd42158700 1 Processor -- start 2024-03-18T01:18:41.106 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:41.100+0000 7fdd42158700 1 -- start start 2024-03-18T01:18:41.106 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:41.100+0000 7fdd42158700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdd3c0fb4d0 0x7fdd3c113a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:18:41.106 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:41.100+0000 7fdd42158700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdd3c102f50 0x7fdd3c113f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:18:41.106 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:41.100+0000 7fdd42158700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd3c1145b0 con 0x7fdd3c0fb4d0 2024-03-18T01:18:41.106 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:41.100+0000 7fdd42158700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd3c10db90 con 0x7fdd3c102f50 2024-03-18T01:18:41.106 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:41.100+0000 7fdd3b7fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdd3c0fb4d0 0x7fdd3c113a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:18:41.106 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:41.100+0000 7fdd3b7fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdd3c0fb4d0 0x7fdd3c113a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:55284/0 (socket says 172.21.15.29:55284) 2024-03-18T01:18:41.106 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:41.100+0000 7fdd3b7fe700 1 -- 172.21.15.29:0/213040238 learned_addr learned my addr 172.21.15.29:0/213040238 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-18T01:18:41.106 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:41.100+0000 7fdd3affd700 1 --2- 172.21.15.29:0/213040238 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdd3c102f50 0x7fdd3c113f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:18:41.106 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:41.100+0000 7fdd3b7fe700 1 -- 172.21.15.29:0/213040238 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdd3c102f50 msgr2=0x7fdd3c113f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:18:41.107 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:41.100+0000 7fdd3b7fe700 1 --2- 172.21.15.29:0/213040238 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdd3c102f50 0x7fdd3c113f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:18:41.107 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:41.100+0000 7fdd3b7fe700 1 -- 172.21.15.29:0/213040238 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd30005040 con 0x7fdd3c0fb4d0 2024-03-18T01:18:41.107 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:41.100+0000 7fdd3affd700 1 --2- 172.21.15.29:0/213040238 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdd3c102f50 0x7fdd3c113f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T01:18:41.107 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:41.101+0000 7fdd3b7fe700 1 --2- 172.21.15.29:0/213040238 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdd3c0fb4d0 0x7fdd3c113a20 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fdd30009180 tx=0x7fdd30009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:18:41.107 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:41.101+0000 7fdd23fff700 1 -- 172.21.15.29:0/213040238 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdd3000e040 con 0x7fdd3c0fb4d0 2024-03-18T01:18:41.107 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:41.101+0000 7fdd23fff700 1 -- 172.21.15.29:0/213040238 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fdd3000bbc0 con 0x7fdd3c0fb4d0 2024-03-18T01:18:41.107 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:41.101+0000 7fdd42158700 1 -- 172.21.15.29:0/213040238 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd3c10de10 con 0x7fdd3c0fb4d0 2024-03-18T01:18:41.107 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:41.101+0000 7fdd23fff700 1 -- 172.21.15.29:0/213040238 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdd3001c3f0 con 0x7fdd3c0fb4d0 2024-03-18T01:18:41.107 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:41.101+0000 7fdd42158700 1 -- 172.21.15.29:0/213040238 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd3c10e350 con 0x7fdd3c0fb4d0 2024-03-18T01:18:41.107 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:41.102+0000 7fdd42158700 1 -- 172.21.15.29:0/213040238 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd3c100200 con 0x7fdd3c0fb4d0 2024-03-18T01:18:41.107 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:41.102+0000 7fdd23fff700 1 -- 172.21.15.29:0/213040238 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7fdd3000bd30 con 0x7fdd3c0fb4d0 2024-03-18T01:18:41.107 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:41.103+0000 7fdd23fff700 1 --2- 172.21.15.29:0/213040238 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7fdd240692a0 0x7fdd2406b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:18:41.107 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:41.103+0000 7fdd23fff700 1 -- 172.21.15.29:0/213040238 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7fdd30088500 con 0x7fdd3c0fb4d0 2024-03-18T01:18:41.107 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:41.103+0000 7fdd3affd700 1 --2- 172.21.15.29:0/213040238 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7fdd240692a0 0x7fdd2406b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:18:41.107 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:41.103+0000 7fdd3affd700 1 --2- 172.21.15.29:0/213040238 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7fdd240692a0 0x7fdd2406b750 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fdd2c007fd0 tx=0x7fdd2c009040 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:18:41.108 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:41.107+0000 7fdd23fff700 1 -- 172.21.15.29:0/213040238 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175520 (secure 0 0 0) 0x7fdd30058250 con 0x7fdd3c0fb4d0 2024-03-18T01:18:41.346 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:41.345+0000 7fdd42158700 1 -- 172.21.15.29:0/213040238 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fdd3c04c710 con 0x7fdd3c0fb4d0 2024-03-18T01:18:41.346 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:41.346+0000 7fdd23fff700 1 -- 172.21.15.29:0/213040238 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v20) v1 ==== 53+0+83 (secure 0 0 0) 0x7fdd30057de0 con 0x7fdd3c0fb4d0 2024-03-18T01:18:41.346 INFO:teuthology.orchestra.run.smithi029.stdout:name: cephfs, metadata pool: cephfs.cephfs.meta, data pools: [cephfs.cephfs.data ] 2024-03-18T01:18:41.348 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:41.348+0000 7fdd42158700 1 -- 172.21.15.29:0/213040238 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7fdd240692a0 msgr2=0x7fdd2406b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:18:41.348 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:41.348+0000 7fdd42158700 1 --2- 172.21.15.29:0/213040238 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7fdd240692a0 0x7fdd2406b750 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fdd2c007fd0 tx=0x7fdd2c009040 comp rx=0 tx=0).stop 2024-03-18T01:18:41.348 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:41.348+0000 7fdd42158700 1 -- 172.21.15.29:0/213040238 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdd3c0fb4d0 msgr2=0x7fdd3c113a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:18:41.348 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:41.348+0000 7fdd42158700 1 --2- 172.21.15.29:0/213040238 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdd3c0fb4d0 0x7fdd3c113a20 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fdd30009180 tx=0x7fdd30009e30 comp rx=0 tx=0).stop 2024-03-18T01:18:41.348 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:41.348+0000 7fdd42158700 1 -- 172.21.15.29:0/213040238 shutdown_connections 2024-03-18T01:18:41.348 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:41.348+0000 7fdd42158700 1 --2- 172.21.15.29:0/213040238 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7fdd240692a0 0x7fdd2406b750 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:18:41.348 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:41.348+0000 7fdd42158700 1 --2- 172.21.15.29:0/213040238 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdd3c0fb4d0 0x7fdd3c113a20 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:18:41.349 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:41.348+0000 7fdd42158700 1 --2- 172.21.15.29:0/213040238 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdd3c102f50 0x7fdd3c113f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:18:41.349 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:41.348+0000 7fdd42158700 1 -- 172.21.15.29:0/213040238 >> 172.21.15.29:0/213040238 conn(0x7fdd3c0714c0 msgr2=0x7fdd3c0fa6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:18:41.349 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:41.348+0000 7fdd42158700 1 -- 172.21.15.29:0/213040238 shutdown_connections 2024-03-18T01:18:41.349 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:41.348+0000 7fdd42158700 1 -- 172.21.15.29:0/213040238 wait complete. 2024-03-18T01:18:41.359 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:18:41 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-18T01:18:41.776 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:18:41 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-18T01:18:42.009 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-18T01:18:42.009 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-18T01:18:42.009 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi090.front.sepia.ceph.com 2024-03-18T01:18:42.009 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-03-18T01:18:42.009 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-03-18T01:18:42.009 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-18T01:18:42.010 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-18T01:18:42.010 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-18T01:18:42.010 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.1' 2024-03-18T01:18:42.010 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T01:18:42.010 DEBUG:teuthology.orchestra.run.smithi090:> ip addr 2024-03-18T01:18:42.030 INFO:teuthology.orchestra.run.smithi090.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-18T01:18:42.030 INFO:teuthology.orchestra.run.smithi090.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-18T01:18:42.030 INFO:teuthology.orchestra.run.smithi090.stdout: inet 127.0.0.1/8 scope host lo 2024-03-18T01:18:42.030 INFO:teuthology.orchestra.run.smithi090.stdout: valid_lft forever preferred_lft forever 2024-03-18T01:18:42.030 INFO:teuthology.orchestra.run.smithi090.stdout: inet6 ::1/128 scope host 2024-03-18T01:18:42.031 INFO:teuthology.orchestra.run.smithi090.stdout: valid_lft forever preferred_lft forever 2024-03-18T01:18:42.031 INFO:teuthology.orchestra.run.smithi090.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-18T01:18:42.031 INFO:teuthology.orchestra.run.smithi090.stdout: link/ether 0c:c4:7a:d8:a1:a8 brd ff:ff:ff:ff:ff:ff 2024-03-18T01:18:42.031 INFO:teuthology.orchestra.run.smithi090.stdout: altname enp7s0f0 2024-03-18T01:18:42.031 INFO:teuthology.orchestra.run.smithi090.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-18T01:18:42.031 INFO:teuthology.orchestra.run.smithi090.stdout: link/ether 0c:c4:7a:d8:a1:a9 brd ff:ff:ff:ff:ff:ff 2024-03-18T01:18:42.031 INFO:teuthology.orchestra.run.smithi090.stdout: altname enp7s0f1 2024-03-18T01:18:42.031 INFO:teuthology.orchestra.run.smithi090.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-18T01:18:42.031 INFO:teuthology.orchestra.run.smithi090.stdout: link/ether 0c:c4:7a:8f:d7:66 brd ff:ff:ff:ff:ff:ff 2024-03-18T01:18:42.031 INFO:teuthology.orchestra.run.smithi090.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-18T01:18:42.031 INFO:teuthology.orchestra.run.smithi090.stdout: link/ether 0c:c4:7a:8f:d7:67 brd ff:ff:ff:ff:ff:ff 2024-03-18T01:18:42.031 INFO:teuthology.orchestra.run.smithi090.stdout: inet 172.21.15.90/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-18T01:18:42.031 INFO:teuthology.orchestra.run.smithi090.stdout: valid_lft 42217sec preferred_lft 42217sec 2024-03-18T01:18:42.031 INFO:teuthology.orchestra.run.smithi090.stdout: inet6 fe80::ec4:7aff:fe8f:d767/64 scope link 2024-03-18T01:18:42.031 INFO:teuthology.orchestra.run.smithi090.stdout: valid_lft forever preferred_lft forever 2024-03-18T01:18:42.032 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-03-18T01:18:42.032 DEBUG:teuthology.orchestra.run.smithi090:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-18T01:18:42.032 DEBUG:teuthology.orchestra.run.smithi090:> set -e 2024-03-18T01:18:42.032 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip link add name ceph-brx type bridge 2024-03-18T01:18:42.032 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip addr flush dev ceph-brx 2024-03-18T01:18:42.032 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip link set ceph-brx up 2024-03-18T01:18:42.032 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-03-18T01:18:42.032 DEBUG:teuthology.orchestra.run.smithi090:> ') 2024-03-18T01:18:42.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:18:42 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-18T01:18:42.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:18:42 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-18T01:18:42.241 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T01:18:42.241 DEBUG:teuthology.orchestra.run.smithi090:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-18T01:18:42.267 INFO:teuthology.orchestra.run.smithi090.stdout:1 2024-03-18T01:18:42.268 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T01:18:42.268 DEBUG:teuthology.orchestra.run.smithi090:> route 2024-03-18T01:18:42.357 INFO:teuthology.orchestra.run.smithi090.stdout:Kernel IP routing table 2024-03-18T01:18:42.357 INFO:teuthology.orchestra.run.smithi090.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-18T01:18:42.357 INFO:teuthology.orchestra.run.smithi090.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-18T01:18:42.357 INFO:teuthology.orchestra.run.smithi090.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-18T01:18:42.357 INFO:teuthology.orchestra.run.smithi090.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-18T01:18:42.358 DEBUG:teuthology.orchestra.run.smithi090:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-18T01:18:42.359 DEBUG:teuthology.orchestra.run.smithi090:> set -e 2024-03-18T01:18:42.359 DEBUG:teuthology.orchestra.run.smithi090:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-18T01:18:42.359 DEBUG:teuthology.orchestra.run.smithi090:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-18T01:18:42.359 DEBUG:teuthology.orchestra.run.smithi090:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-18T01:18:42.359 DEBUG:teuthology.orchestra.run.smithi090:> ') 2024-03-18T01:18:42.406 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:18:42 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-18T01:18:42.417 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:18:42 smithi090 ceph-mon[36989]: pgmap v199: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 170 B/s wr, 4 op/s 2024-03-18T01:18:42.417 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:18:42 smithi090 ceph-mon[36989]: from='client.? 172.21.15.29:0/213040238' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-18T01:18:42.612 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:18:42 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-18T01:18:42.618 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T01:18:42.618 DEBUG:teuthology.orchestra.run.smithi090:> ip netns list 2024-03-18T01:18:42.633 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T01:18:42.633 DEBUG:teuthology.orchestra.run.smithi090:> ip netns list-id 2024-03-18T01:18:42.688 DEBUG:teuthology.orchestra.run.smithi090:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-18T01:18:42.688 DEBUG:teuthology.orchestra.run.smithi090:> set -e 2024-03-18T01:18:42.688 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-18T01:18:42.688 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.1 0 2024-03-18T01:18:42.688 DEBUG:teuthology.orchestra.run.smithi090:> ') 2024-03-18T01:18:42.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:18:42 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-18T01:18:42.805 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:18:42 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-18T01:18:42.811 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.1' with 192.168.0.1/16 2024-03-18T01:18:42.811 DEBUG:teuthology.orchestra.run.smithi090:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-18T01:18:42.811 DEBUG:teuthology.orchestra.run.smithi090:> set -e 2024-03-18T01:18:42.812 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.1 type veth peer name brx.0 2024-03-18T01:18:42.812 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-03-18T01:18:42.812 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set veth0 up 2024-03-18T01:18:42.812 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set lo up 2024-03-18T01:18:42.812 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip route add default via 192.168.255.254 2024-03-18T01:18:42.812 DEBUG:teuthology.orchestra.run.smithi090:> ') 2024-03-18T01:18:42.819 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:18:42 smithi029 ceph-mon[29331]: pgmap v199: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 170 B/s wr, 4 op/s 2024-03-18T01:18:42.819 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:18:42 smithi029 ceph-mon[29331]: from='client.? 172.21.15.29:0/213040238' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-18T01:18:42.859 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:18:42 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-18T01:18:42.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:18:42 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-18T01:18:42.949 DEBUG:teuthology.orchestra.run.smithi090:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-18T01:18:42.949 DEBUG:teuthology.orchestra.run.smithi090:> set -e 2024-03-18T01:18:42.949 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip link set brx.0 up 2024-03-18T01:18:42.949 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip link set dev brx.0 master ceph-brx 2024-03-18T01:18:42.949 DEBUG:teuthology.orchestra.run.smithi090:> ') 2024-03-18T01:18:42.996 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:18:42 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-18T01:18:43.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:18:43 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-18T01:18:43.029 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-18T01:18:43.029 DEBUG:teuthology.orchestra.run.smithi090:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-03-18T01:18:43.044 INFO:teuthology.orchestra.run.smithi090.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-03-18T01:18:43.044 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-18T01:18:43.044 DEBUG:teuthology.orchestra.run.smithi090:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-03-18T01:18:43.099 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-03-18T01:18:43.099 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T01:18:43.099 DEBUG:teuthology.orchestra.run.smithi090:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.1 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-03-18T01:18:43.203 INFO:teuthology.orchestra.run.smithi090.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-03-18T01:18:43.203 INFO:teuthology.orchestra.run.smithi090.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2024-03-18T01:18:43.203 INFO:teuthology.orchestra.run.smithi090.stdout:invalid new device string format 2024-03-18T01:18:43.327 INFO:teuthology.orchestra.run.smithi090.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-03-18T01:18:43.328 INFO:teuthology.orchestra.run.smithi090.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2024-03-18T01:18:43.328 INFO:teuthology.orchestra.run.smithi090.stdout:invalid new device string format 2024-03-18T01:18:43.328 INFO:teuthology.orchestra.run.smithi090.stdout:mount.ceph: resolved to: "172.21.15.29:3300,172.21.15.90:3300" 2024-03-18T01:18:43.328 INFO:teuthology.orchestra.run.smithi090.stdout:mount.ceph: switching to using v1 address with old syntax 2024-03-18T01:18:43.330 INFO:teuthology.orchestra.run.smithi090.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-03-18T01:18:43.330 INFO:teuthology.orchestra.run.smithi090.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2024-03-18T01:18:43.330 INFO:teuthology.orchestra.run.smithi090.stdout:invalid new device string format 2024-03-18T01:18:43.330 INFO:teuthology.orchestra.run.smithi090.stdout:mount.ceph: resolved to: "172.21.15.29:3300,172.21.15.90:3300" 2024-03-18T01:18:43.330 INFO:teuthology.orchestra.run.smithi090.stdout:mount.ceph: switching to using v1 address with old syntax 2024-03-18T01:18:43.330 INFO:teuthology.orchestra.run.smithi090.stdout:mount.ceph: resolved to: "172.21.15.29:6789,172.21.15.90:6789" 2024-03-18T01:18:43.331 INFO:teuthology.orchestra.run.smithi090.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.29:6789,172.21.15.90:6789:/ 2024-03-18T01:18:43.331 INFO:teuthology.orchestra.run.smithi090.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs,key=1,fsid=37f51182-e4c4-11ee-95c9-87774f69a715" will pass to kernel 2024-03-18T01:18:43.331 INFO:teuthology.orchestra.run.smithi090.stdout:mount: /home/ubuntu/cephtest/mnt.1 does not contain SELinux labels. 2024-03-18T01:18:43.331 INFO:teuthology.orchestra.run.smithi090.stdout: You just mounted a file system that supports labels which does not 2024-03-18T01:18:43.331 INFO:teuthology.orchestra.run.smithi090.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-03-18T01:18:43.331 INFO:teuthology.orchestra.run.smithi090.stdout: applications will generate AVC messages and not be allowed access to 2024-03-18T01:18:43.331 INFO:teuthology.orchestra.run.smithi090.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-03-18T01:18:43.333 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-03-18T01:18:43.333 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T01:18:43.333 DEBUG:teuthology.orchestra.run.smithi090:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-03-18T01:18:43.360 DEBUG:teuthology.orchestra.run.smithi090:> (cd /home/ubuntu/cephtest/mnt.1 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-18T01:18:43.449 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:18:43 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-03-18T01:18:43.450 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:18:43 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-18T01:18:43.456 INFO:teuthology.orchestra.run.smithi090.stdout:client14538 2024-03-18T01:18:43.456 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-03-18T01:18:43.456 DEBUG:teuthology.orchestra.run.smithi090:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-18T01:18:43.472 DEBUG:teuthology.orchestra.run.smithi090:> (cd /home/ubuntu/cephtest/mnt.1 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-18T01:18:43.525 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:18:43 smithi090 ceph-mon[36989]: pgmap v200: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 170 B/s wr, 4 op/s 2024-03-18T01:18:43.559 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:18:43 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-03-18T01:18:43.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:18:43 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-18T01:18:43.567 INFO:teuthology.orchestra.run.smithi090.stdout:client14538 2024-03-18T01:18:43.567 DEBUG:teuthology.orchestra.run.smithi090:> (cd /home/ubuntu/cephtest/mnt.1 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-18T01:18:43.654 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:18:43 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-03-18T01:18:43.656 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:18:43 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-18T01:18:43.661 INFO:teuthology.orchestra.run.smithi090.stdout:client14538 2024-03-18T01:18:43.661 DEBUG:teuthology.orchestra.run.smithi090:> (cd /home/ubuntu/cephtest/mnt.1 && exec stdin-killer --timeout=300 -- bash -c 'sudo dd if=/sys/kernel/debug/ceph/37f51182-e4c4-11ee-95c9-87774f69a715.client14538/status') 2024-03-18T01:18:43.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:18:43 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-18T01:18:43.721 INFO:teuthology.orchestra.run.smithi090.stdout:instance: client.14538 (3)172.21.15.90:0/228512458 2024-03-18T01:18:43.721 INFO:teuthology.orchestra.run.smithi090.stdout:blocklisted: false 2024-03-18T01:18:43.721 INFO:teuthology.orchestra.run.smithi090.stderr:0+1 records in 2024-03-18T01:18:43.721 INFO:teuthology.orchestra.run.smithi090.stderr:0+1 records out 2024-03-18T01:18:43.722 INFO:teuthology.orchestra.run.smithi090.stderr:70 bytes copied, 2.0973e-05 s, 3.3 MB/s 2024-03-18T01:18:43.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-18T01:18:43 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-18T01:18:43.728 INFO:teuthology.run_tasks:Running task print... 2024-03-18T01:18:43.739 INFO:teuthology.task.print:**** done client 2024-03-18T01:18:43.739 INFO:teuthology.run_tasks:Running task parallel... 2024-03-18T01:18:43.758 INFO:teuthology.task.parallel:starting parallel... 2024-03-18T01:18:43.759 INFO:teuthology.task.parallel:In parallel, running task sequential... 2024-03-18T01:18:43.767 INFO:teuthology.task.sequential:In sequential, running task cephadm.shell... 2024-03-18T01:18:43.767 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi029.front.sepia.ceph.com 2024-03-18T01:18:43.767 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 37f51182-e4c4-11ee-95c9-87774f69a715 -e sha1=3d78f5ba07bf18880172c07d4d7b4ac00c9d30af -- bash -c 'ceph config set mgr mgr/orchestrator/fail_fs false || true' 2024-03-18T01:18:43.768 INFO:teuthology.task.parallel:In parallel, running task sequential... 2024-03-18T01:18:43.768 INFO:teuthology.task.sequential:In sequential, running task workunit... 2024-03-18T01:18:43.771 INFO:tasks.workunit:Pulling workunits from ref c382027d5c66ef4e0aa1131ea3e8080c658d3184 2024-03-18T01:18:43.772 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-03-18T01:18:43.772 INFO:tasks.workunit:timeout=3h 2024-03-18T01:18:43.772 INFO:tasks.workunit:cleanup=True 2024-03-18T01:18:43.772 DEBUG:teuthology.orchestra.run.smithi029:> stat -- /home/ubuntu/cephtest/mnt.0 2024-03-18T01:18:43.792 INFO:teuthology.orchestra.run.smithi029.stdout: File: /home/ubuntu/cephtest/mnt.0 2024-03-18T01:18:43.792 INFO:teuthology.orchestra.run.smithi029.stdout: Size: 0 Blocks: 0 IO Block: 65536 directory 2024-03-18T01:18:43.793 INFO:teuthology.orchestra.run.smithi029.stdout:Device: 94h/148d Inode: 1 Links: 2 2024-03-18T01:18:43.793 INFO:teuthology.orchestra.run.smithi029.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-18T01:18:43.793 INFO:teuthology.orchestra.run.smithi029.stdout:Context: system_u:object_r:unlabeled_t:s0 2024-03-18T01:18:43.793 INFO:teuthology.orchestra.run.smithi029.stdout:Access: 1970-01-01 00:00:00.000000000 +0000 2024-03-18T01:18:43.793 INFO:teuthology.orchestra.run.smithi029.stdout:Modify: 2024-03-18 01:18:07.248053465 +0000 2024-03-18T01:18:43.793 INFO:teuthology.orchestra.run.smithi029.stdout:Change: 2024-03-18 01:18:38.672278808 +0000 2024-03-18T01:18:43.793 INFO:teuthology.orchestra.run.smithi029.stdout: Birth: 2024-03-18 01:18:07.248053465 +0000 2024-03-18T01:18:43.793 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.0 2024-03-18T01:18:43.793 DEBUG:teuthology.orchestra.run.smithi029:> cd -- /home/ubuntu/cephtest/mnt.0 && sudo install -d -m 0755 --owner=ubuntu -- client.0 2024-03-18T01:18:43.803 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:18:43 smithi029 ceph-mon[29331]: pgmap v200: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 170 B/s wr, 4 op/s 2024-03-18T01:18:43.837 DEBUG:teuthology.orchestra.run.smithi090:> stat -- /home/ubuntu/cephtest/mnt.1 2024-03-18T01:18:43.854 INFO:teuthology.orchestra.run.smithi090.stdout: File: /home/ubuntu/cephtest/mnt.1 2024-03-18T01:18:43.854 INFO:teuthology.orchestra.run.smithi090.stdout: Size: 1 Blocks: 0 IO Block: 65536 directory 2024-03-18T01:18:43.854 INFO:teuthology.orchestra.run.smithi090.stdout:Device: 73h/115d Inode: 1 Links: 3 2024-03-18T01:18:43.854 INFO:teuthology.orchestra.run.smithi090.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-18T01:18:43.854 INFO:teuthology.orchestra.run.smithi090.stdout:Context: system_u:object_r:unlabeled_t:s0 2024-03-18T01:18:43.854 INFO:teuthology.orchestra.run.smithi090.stdout:Access: 1970-01-01 00:00:00.000000000 +0000 2024-03-18T01:18:43.854 INFO:teuthology.orchestra.run.smithi090.stdout:Modify: 2024-03-18 01:18:43.831192666 +0000 2024-03-18T01:18:43.854 INFO:teuthology.orchestra.run.smithi090.stdout:Change: 2024-03-18 01:18:43.831192666 +0000 2024-03-18T01:18:43.854 INFO:teuthology.orchestra.run.smithi090.stdout: Birth: 2024-03-18 01:18:07.248053465 +0000 2024-03-18T01:18:43.855 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.1 2024-03-18T01:18:43.855 DEBUG:teuthology.orchestra.run.smithi090:> cd -- /home/ubuntu/cephtest/mnt.1 && sudo install -d -m 0755 --owner=ubuntu -- client.1 2024-03-18T01:18:43.926 DEBUG:teuthology.orchestra.run.smithi029:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://github.com/batrick/ceph /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout c382027d5c66ef4e0aa1131ea3e8080c658d3184 2024-03-18T01:18:43.927 DEBUG:teuthology.orchestra.run.smithi090:> rm -rf /home/ubuntu/cephtest/clone.client.1 && git clone https://github.com/batrick/ceph /home/ubuntu/cephtest/clone.client.1 && cd /home/ubuntu/cephtest/clone.client.1 && git checkout c382027d5c66ef4e0aa1131ea3e8080c658d3184 2024-03-18T01:18:43.942 INFO:tasks.workunit.client.0.smithi029.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-03-18T01:18:43.982 INFO:tasks.workunit.client.1.smithi090.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.1'... 2024-03-18T01:18:44.057 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/37f51182-e4c4-11ee-95c9-87774f69a715/mon.smithi029/config 2024-03-18T01:18:45.933 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:45.889+0000 7f19863c1700 1 -- 172.21.15.29:0/2291840347 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f19800fcef0 msgr2=0x7f19800fd360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:18:45.933 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:45.889+0000 7f19863c1700 1 --2- 172.21.15.29:0/2291840347 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f19800fcef0 0x7f19800fd360 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f1974004940 tx=0x7f1974009e30 comp rx=0 tx=0).stop 2024-03-18T01:18:45.933 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:45.889+0000 7f19863c1700 1 -- 172.21.15.29:0/2291840347 shutdown_connections 2024-03-18T01:18:45.933 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:45.889+0000 7f19863c1700 1 --2- 172.21.15.29:0/2291840347 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f19800fcef0 0x7f19800fd360 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:18:45.934 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:45.889+0000 7f19863c1700 1 --2- 172.21.15.29:0/2291840347 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f19800fbc40 0x7f19800fc050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:18:45.934 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:45.889+0000 7f19863c1700 1 -- 172.21.15.29:0/2291840347 >> 172.21.15.29:0/2291840347 conn(0x7f19800f73c0 msgr2=0x7f19800f97d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:18:45.934 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:45.889+0000 7f19863c1700 1 -- 172.21.15.29:0/2291840347 shutdown_connections 2024-03-18T01:18:45.934 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:45.889+0000 7f19863c1700 1 -- 172.21.15.29:0/2291840347 wait complete. 2024-03-18T01:18:45.934 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:45.889+0000 7f19863c1700 1 Processor -- start 2024-03-18T01:18:45.934 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:45.889+0000 7f19863c1700 1 -- start start 2024-03-18T01:18:45.934 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:45.889+0000 7f19863c1700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f19800fbc40 0x7f19800ffd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:18:45.934 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:45.890+0000 7f19863c1700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f19800fcef0 0x7f19800fe230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:18:45.935 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:45.890+0000 7f19863c1700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19800fe7e0 con 0x7f19800fbc40 2024-03-18T01:18:45.935 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:45.890+0000 7f19863c1700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19800fe950 con 0x7f19800fcef0 2024-03-18T01:18:45.935 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:45.890+0000 7f197ffff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f19800fbc40 0x7f19800ffd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:18:45.935 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:45.890+0000 7f197ffff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f19800fbc40 0x7f19800ffd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56736/0 (socket says 172.21.15.29:56736) 2024-03-18T01:18:45.935 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:45.890+0000 7f197ffff700 1 -- 172.21.15.29:0/603184826 learned_addr learned my addr 172.21.15.29:0/603184826 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-18T01:18:45.935 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:45.890+0000 7f197f7fe700 1 --2- 172.21.15.29:0/603184826 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f19800fcef0 0x7f19800fe230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:18:45.935 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:45.890+0000 7f197ffff700 1 -- 172.21.15.29:0/603184826 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f19800fcef0 msgr2=0x7f19800fe230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:18:45.935 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:45.890+0000 7f197ffff700 1 --2- 172.21.15.29:0/603184826 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f19800fcef0 0x7f19800fe230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:18:45.935 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:45.890+0000 7f197ffff700 1 -- 172.21.15.29:0/603184826 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1974005040 con 0x7f19800fbc40 2024-03-18T01:18:45.936 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:45.890+0000 7f197ffff700 1 --2- 172.21.15.29:0/603184826 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f19800fbc40 0x7f19800ffd60 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f197000c8c0 tx=0x7f19700044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:18:45.936 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:45.890+0000 7f197cff9700 1 -- 172.21.15.29:0/603184826 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f197000a760 con 0x7f19800fbc40 2024-03-18T01:18:45.936 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:45.890+0000 7f197cff9700 1 -- 172.21.15.29:0/603184826 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f197000ad90 con 0x7f19800fbc40 2024-03-18T01:18:45.936 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:45.890+0000 7f197cff9700 1 -- 172.21.15.29:0/603184826 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f19700113a0 con 0x7f19800fbc40 2024-03-18T01:18:45.936 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:45.890+0000 7f19863c1700 1 -- 172.21.15.29:0/603184826 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19800fec30 con 0x7f19800fbc40 2024-03-18T01:18:45.936 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:45.891+0000 7f19863c1700 1 -- 172.21.15.29:0/603184826 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1980115920 con 0x7f19800fbc40 2024-03-18T01:18:45.936 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:45.892+0000 7f19863c1700 1 -- 172.21.15.29:0/603184826 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f198005f5f0 con 0x7f19800fbc40 2024-03-18T01:18:45.936 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:45.892+0000 7f197cff9700 1 -- 172.21.15.29:0/603184826 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f1970012400 con 0x7f19800fbc40 2024-03-18T01:18:45.936 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:45.892+0000 7f197cff9700 1 --2- 172.21.15.29:0/603184826 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f19680692a0 0x7f196806b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:18:45.936 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:45.892+0000 7f197cff9700 1 -- 172.21.15.29:0/603184826 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f19700871f0 con 0x7f19800fbc40 2024-03-18T01:18:45.937 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:45.893+0000 7f197f7fe700 1 --2- 172.21.15.29:0/603184826 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f19680692a0 0x7f196806b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:18:45.937 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:45.893+0000 7f197f7fe700 1 --2- 172.21.15.29:0/603184826 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f19680692a0 0x7f196806b750 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f197400b3b0 tx=0x7f1974009e30 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:18:45.937 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:45.897+0000 7f197cff9700 1 -- 172.21.15.29:0/603184826 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175520 (secure 0 0 0) 0x7f19700534d0 con 0x7f19800fbc40 2024-03-18T01:18:46.094 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:46.093+0000 7f19863c1700 1 -- 172.21.15.29:0/603184826 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command([{prefix=config set, name=mgr/orchestrator/fail_fs}] v 0) v1 -- 0x7f198004bed0 con 0x7f19800fbc40 2024-03-18T01:18:46.094 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:46.094+0000 7f197cff9700 1 -- 172.21.15.29:0/603184826 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/orchestrator/fail_fs}]=-22 unrecognized config option 'mgr/orchestrator/fail_fs' v21)=-22 unrecognized config option 'mgr/orchestrator/fail_fs' v21) v1 ==== 179+0+0 (secure 0 0 0) 0x7f1970056af0 con 0x7f19800fbc40 2024-03-18T01:18:46.096 INFO:teuthology.orchestra.run.smithi029.stderr:Error EINVAL: unrecognized config option 'mgr/orchestrator/fail_fs' 2024-03-18T01:18:46.457 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:18:46 smithi029 ceph-mon[29331]: pgmap v201: 65 pgs: 65 active+clean; 585 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.2 KiB/s rd, 682 B/s wr, 4 op/s 2024-03-18T01:18:46.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:18:46 smithi090 ceph-mon[36989]: pgmap v201: 65 pgs: 65 active+clean; 585 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.2 KiB/s rd, 682 B/s wr, 4 op/s 2024-03-18T01:18:46.801 INFO:teuthology.task.sequential:In sequential, running task cephadm.shell... 2024-03-18T01:18:46.802 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi029.front.sepia.ceph.com 2024-03-18T01:18:46.802 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 37f51182-e4c4-11ee-95c9-87774f69a715 -e sha1=3d78f5ba07bf18880172c07d4d7b4ac00c9d30af -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force' 2024-03-18T01:18:47.123 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/37f51182-e4c4-11ee-95c9-87774f69a715/mon.smithi029/config 2024-03-18T01:18:48.319 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:18:48 smithi029 ceph-mon[29331]: pgmap v202: 65 pgs: 65 active+clean; 586 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.9 KiB/s rd, 853 B/s wr, 4 op/s 2024-03-18T01:18:48.525 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:18:48 smithi090 ceph-mon[36989]: pgmap v202: 65 pgs: 65 active+clean; 586 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.9 KiB/s rd, 853 B/s wr, 4 op/s 2024-03-18T01:18:49.016 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:49.013+0000 7fa7d6807700 1 -- 172.21.15.29:0/2974472952 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa7d00fe440 msgr2=0x7fa7d00fe850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:18:49.017 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:49.013+0000 7fa7d6807700 1 --2- 172.21.15.29:0/2974472952 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa7d00fe440 0x7fa7d00fe850 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fa7c0004940 tx=0x7fa7c0009e30 comp rx=0 tx=0).stop 2024-03-18T01:18:49.017 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:49.013+0000 7fa7d6807700 1 -- 172.21.15.29:0/2974472952 shutdown_connections 2024-03-18T01:18:49.017 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:49.013+0000 7fa7d6807700 1 --2- 172.21.15.29:0/2974472952 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa7d00fed90 0x7fa7d00faed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:18:49.017 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:49.013+0000 7fa7d6807700 1 --2- 172.21.15.29:0/2974472952 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa7d00fe440 0x7fa7d00fe850 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:18:49.018 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:49.013+0000 7fa7d6807700 1 -- 172.21.15.29:0/2974472952 >> 172.21.15.29:0/2974472952 conn(0x7fa7d00f6bc0 msgr2=0x7fa7d00f9010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:18:49.018 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:49.013+0000 7fa7d6807700 1 -- 172.21.15.29:0/2974472952 shutdown_connections 2024-03-18T01:18:49.018 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:49.013+0000 7fa7d6807700 1 -- 172.21.15.29:0/2974472952 wait complete. 2024-03-18T01:18:49.018 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:49.014+0000 7fa7d6807700 1 Processor -- start 2024-03-18T01:18:49.018 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:49.014+0000 7fa7d6807700 1 -- start start 2024-03-18T01:18:49.018 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:49.014+0000 7fa7d6807700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa7d00fe440 0x7fa7d010d8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:18:49.019 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:49.014+0000 7fa7d6807700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa7d00fed90 0x7fa7d010de20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:18:49.019 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:49.014+0000 7fa7d6807700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7d010e420 con 0x7fa7d00fed90 2024-03-18T01:18:49.019 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:49.014+0000 7fa7d6807700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7d010e590 con 0x7fa7d00fe440 2024-03-18T01:18:49.019 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:49.014+0000 7fa7c7fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa7d00fed90 0x7fa7d010de20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:18:49.019 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:49.014+0000 7fa7c7fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa7d00fed90 0x7fa7d010de20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56754/0 (socket says 172.21.15.29:56754) 2024-03-18T01:18:49.019 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:49.014+0000 7fa7c7fff700 1 -- 172.21.15.29:0/3699555284 learned_addr learned my addr 172.21.15.29:0/3699555284 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-18T01:18:49.019 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:49.014+0000 7fa7cffff700 1 --2- 172.21.15.29:0/3699555284 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa7d00fe440 0x7fa7d010d8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:18:49.020 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:49.015+0000 7fa7c7fff700 1 -- 172.21.15.29:0/3699555284 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa7d00fe440 msgr2=0x7fa7d010d8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:18:49.020 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:49.015+0000 7fa7c7fff700 1 --2- 172.21.15.29:0/3699555284 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa7d00fe440 0x7fa7d010d8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:18:49.020 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:49.015+0000 7fa7c7fff700 1 -- 172.21.15.29:0/3699555284 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7c0005040 con 0x7fa7d00fed90 2024-03-18T01:18:49.020 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:49.015+0000 7fa7c7fff700 1 --2- 172.21.15.29:0/3699555284 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa7d00fed90 0x7fa7d010de20 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fa7b8002a80 tx=0x7fa7b80046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:18:49.020 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:49.015+0000 7fa7cd7fa700 1 -- 172.21.15.29:0/3699555284 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa7b8007cc0 con 0x7fa7d00fed90 2024-03-18T01:18:49.020 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:49.015+0000 7fa7cd7fa700 1 -- 172.21.15.29:0/3699555284 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa7b800f040 con 0x7fa7d00fed90 2024-03-18T01:18:49.021 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:49.015+0000 7fa7d6807700 1 -- 172.21.15.29:0/3699555284 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7d0112ff0 con 0x7fa7d00fed90 2024-03-18T01:18:49.021 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:49.015+0000 7fa7cd7fa700 1 -- 172.21.15.29:0/3699555284 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa7b800dac0 con 0x7fa7d00fed90 2024-03-18T01:18:49.022 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:49.015+0000 7fa7d6807700 1 -- 172.21.15.29:0/3699555284 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7d0113580 con 0x7fa7d00fed90 2024-03-18T01:18:49.022 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:49.016+0000 7fa7d6807700 1 -- 172.21.15.29:0/3699555284 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa7d004c710 con 0x7fa7d00fed90 2024-03-18T01:18:49.025 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:49.017+0000 7fa7cd7fa700 1 -- 172.21.15.29:0/3699555284 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7fa7b800dc20 con 0x7fa7d00fed90 2024-03-18T01:18:49.025 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:49.017+0000 7fa7cd7fa700 1 --2- 172.21.15.29:0/3699555284 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7fa7bc0692a0 0x7fa7bc06b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:18:49.025 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:49.017+0000 7fa7cd7fa700 1 -- 172.21.15.29:0/3699555284 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7fa7b8086f60 con 0x7fa7d00fed90 2024-03-18T01:18:49.025 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:49.017+0000 7fa7cffff700 1 --2- 172.21.15.29:0/3699555284 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7fa7bc0692a0 0x7fa7bc06b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:18:49.025 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:49.018+0000 7fa7cffff700 1 --2- 172.21.15.29:0/3699555284 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7fa7bc0692a0 0x7fa7bc06b750 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fa7c000b3b0 tx=0x7fa7c0009e30 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:18:49.025 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:49.022+0000 7fa7cd7fa700 1 -- 172.21.15.29:0/3699555284 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175520 (secure 0 0 0) 0x7fa7b8056c50 con 0x7fa7d00fed90 2024-03-18T01:18:49.260 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:49.260+0000 7fa7d6807700 1 -- 172.21.15.29:0/3699555284 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim}] v 0) v1 -- 0x7fa7d0113a40 con 0x7fa7d00fed90 2024-03-18T01:18:49.261 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:49.260+0000 7fa7cd7fa700 1 -- 172.21.15.29:0/3699555284 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim}]=0 v21)=0 v21) v1 ==== 155+0+0 (secure 0 0 0) 0x7fa7b80567e0 con 0x7fa7d00fed90 2024-03-18T01:18:49.264 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:49.263+0000 7fa7d6807700 1 -- 172.21.15.29:0/3699555284 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7fa7bc0692a0 msgr2=0x7fa7bc06b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:18:49.264 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:49.263+0000 7fa7d6807700 1 --2- 172.21.15.29:0/3699555284 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7fa7bc0692a0 0x7fa7bc06b750 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fa7c000b3b0 tx=0x7fa7c0009e30 comp rx=0 tx=0).stop 2024-03-18T01:18:49.264 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:49.263+0000 7fa7d6807700 1 -- 172.21.15.29:0/3699555284 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa7d00fed90 msgr2=0x7fa7d010de20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:18:49.264 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:49.263+0000 7fa7d6807700 1 --2- 172.21.15.29:0/3699555284 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa7d00fed90 0x7fa7d010de20 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fa7b8002a80 tx=0x7fa7b80046b0 comp rx=0 tx=0).stop 2024-03-18T01:18:49.264 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:49.263+0000 7fa7d6807700 1 -- 172.21.15.29:0/3699555284 shutdown_connections 2024-03-18T01:18:49.264 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:49.263+0000 7fa7d6807700 1 --2- 172.21.15.29:0/3699555284 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7fa7bc0692a0 0x7fa7bc06b750 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:18:49.265 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:49.263+0000 7fa7d6807700 1 --2- 172.21.15.29:0/3699555284 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa7d00fe440 0x7fa7d010d8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:18:49.265 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:49.263+0000 7fa7d6807700 1 --2- 172.21.15.29:0/3699555284 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa7d00fed90 0x7fa7d010de20 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:18:49.265 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:49.263+0000 7fa7d6807700 1 -- 172.21.15.29:0/3699555284 >> 172.21.15.29:0/3699555284 conn(0x7fa7d00f6bc0 msgr2=0x7fa7d00f8210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:18:49.265 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:49.264+0000 7fa7d6807700 1 -- 172.21.15.29:0/3699555284 shutdown_connections 2024-03-18T01:18:49.265 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:49.264+0000 7fa7d6807700 1 -- 172.21.15.29:0/3699555284 wait complete. 2024-03-18T01:18:50.047 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 37f51182-e4c4-11ee-95c9-87774f69a715 -e sha1=3d78f5ba07bf18880172c07d4d7b4ac00c9d30af -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force' 2024-03-18T01:18:50.319 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:18:50 smithi029 ceph-mon[29331]: pgmap v203: 65 pgs: 65 active+clean; 586 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.3 KiB/s rd, 938 B/s wr, 4 op/s 2024-03-18T01:18:50.321 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/37f51182-e4c4-11ee-95c9-87774f69a715/mon.smithi029/config 2024-03-18T01:18:50.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:18:50 smithi090 ceph-mon[36989]: pgmap v203: 65 pgs: 65 active+clean; 586 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.3 KiB/s rd, 938 B/s wr, 4 op/s 2024-03-18T01:18:52.242 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:18:52 smithi029 ceph-mon[29331]: pgmap v204: 65 pgs: 65 active+clean; 586 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.7 KiB/s rd, 938 B/s wr, 5 op/s 2024-03-18T01:18:52.254 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:52.254+0000 7f1003f95700 1 -- 172.21.15.29:0/88150054 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0ffc0fcf80 msgr2=0x7f0ffc0fd3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:18:52.255 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:52.254+0000 7f1003f95700 1 --2- 172.21.15.29:0/88150054 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0ffc0fcf80 0x7f0ffc0fd3f0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f0fec004940 tx=0x7f0fec009e30 comp rx=0 tx=0).stop 2024-03-18T01:18:52.255 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:52.254+0000 7f1003f95700 1 -- 172.21.15.29:0/88150054 shutdown_connections 2024-03-18T01:18:52.255 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:52.254+0000 7f1003f95700 1 --2- 172.21.15.29:0/88150054 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0ffc0fcf80 0x7f0ffc0fd3f0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:18:52.255 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:52.254+0000 7f1003f95700 1 --2- 172.21.15.29:0/88150054 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0ffc0fbcd0 0x7f0ffc0fc0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:18:52.255 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:52.254+0000 7f1003f95700 1 -- 172.21.15.29:0/88150054 >> 172.21.15.29:0/88150054 conn(0x7f0ffc0f7430 msgr2=0x7f0ffc0f9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:18:52.255 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:52.254+0000 7f1003f95700 1 -- 172.21.15.29:0/88150054 shutdown_connections 2024-03-18T01:18:52.255 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:52.254+0000 7f1003f95700 1 -- 172.21.15.29:0/88150054 wait complete. 2024-03-18T01:18:52.256 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:52.255+0000 7f1003f95700 1 Processor -- start 2024-03-18T01:18:52.256 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:52.255+0000 7f1003f95700 1 -- start start 2024-03-18T01:18:52.257 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:52.255+0000 7f1003f95700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0ffc0fbcd0 0x7f0ffc10c8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:18:52.257 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:52.255+0000 7f1003f95700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0ffc0fcf80 0x7f0ffc10ce00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:18:52.257 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:52.255+0000 7f1003f95700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ffc10d400 con 0x7f0ffc0fbcd0 2024-03-18T01:18:52.257 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:52.255+0000 7f1003f95700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ffc10dc50 con 0x7f0ffc0fcf80 2024-03-18T01:18:52.257 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:52.255+0000 7f1001d31700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0ffc0fbcd0 0x7f0ffc10c8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:18:52.257 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:52.255+0000 7f1001530700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0ffc0fcf80 0x7f0ffc10ce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:18:52.257 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:52.255+0000 7f1001d31700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0ffc0fbcd0 0x7f0ffc10c8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56768/0 (socket says 172.21.15.29:56768) 2024-03-18T01:18:52.257 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:52.255+0000 7f1001d31700 1 -- 172.21.15.29:0/806820501 learned_addr learned my addr 172.21.15.29:0/806820501 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-18T01:18:52.257 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:52.256+0000 7f1001530700 1 -- 172.21.15.29:0/806820501 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0ffc0fbcd0 msgr2=0x7f0ffc10c8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:18:52.257 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:52.256+0000 7f1001530700 1 --2- 172.21.15.29:0/806820501 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0ffc0fbcd0 0x7f0ffc10c8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:18:52.257 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:52.256+0000 7f1001530700 1 -- 172.21.15.29:0/806820501 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fec005040 con 0x7f0ffc0fcf80 2024-03-18T01:18:52.257 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:52.256+0000 7f1001530700 1 --2- 172.21.15.29:0/806820501 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0ffc0fcf80 0x7f0ffc10ce00 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f0fec000c00 tx=0x7f0fec009e30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:18:52.257 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:52.256+0000 7f0ff27fc700 1 -- 172.21.15.29:0/806820501 <== mon.1 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0fec00e030 con 0x7f0ffc0fcf80 2024-03-18T01:18:52.257 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:52.256+0000 7f1003f95700 1 -- 172.21.15.29:0/806820501 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ffc10de50 con 0x7f0ffc0fcf80 2024-03-18T01:18:52.257 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:52.256+0000 7f0ff27fc700 1 -- 172.21.15.29:0/806820501 <== mon.1 v2:172.21.15.90:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0fec01c400 con 0x7f0ffc0fcf80 2024-03-18T01:18:52.258 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:52.256+0000 7f0ff27fc700 1 -- 172.21.15.29:0/806820501 <== mon.1 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0fec00b650 con 0x7f0ffc0fcf80 2024-03-18T01:18:52.258 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:52.256+0000 7f1003f95700 1 -- 172.21.15.29:0/806820501 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ffc10e390 con 0x7f0ffc0fcf80 2024-03-18T01:18:52.258 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:52.257+0000 7f1003f95700 1 -- 172.21.15.29:0/806820501 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ffc05f5f0 con 0x7f0ffc0fcf80 2024-03-18T01:18:52.259 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:52.258+0000 7f0ff27fc700 1 -- 172.21.15.29:0/806820501 <== mon.1 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f0fec013af0 con 0x7f0ffc0fcf80 2024-03-18T01:18:52.259 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:52.258+0000 7f0ff27fc700 1 --2- 172.21.15.29:0/806820501 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f0fe80692a0 0x7f0fe806b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:18:52.259 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:52.258+0000 7f1001d31700 1 --2- 172.21.15.29:0/806820501 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f0fe80692a0 0x7f0fe806b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:18:52.259 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:52.258+0000 7f0ff27fc700 1 -- 172.21.15.29:0/806820501 <== mon.1 v2:172.21.15.90:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f0fec088bd0 con 0x7f0ffc0fcf80 2024-03-18T01:18:52.259 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:52.258+0000 7f1001d31700 1 --2- 172.21.15.29:0/806820501 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f0fe80692a0 0x7f0fe806b750 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f0ff8004bc0 tx=0x7f0ff800b040 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:18:52.262 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:52.262+0000 7f0ff27fc700 1 -- 172.21.15.29:0/806820501 <== mon.1 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175520 (secure 0 0 0) 0x7f0fec058960 con 0x7f0ffc0fcf80 2024-03-18T01:18:52.457 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:18:52 smithi090 ceph-mon[36989]: pgmap v204: 65 pgs: 65 active+clean; 586 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.7 KiB/s rd, 938 B/s wr, 5 op/s 2024-03-18T01:18:52.498 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:52.496+0000 7f1003f95700 1 -- 172.21.15.29:0/806820501 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim_allowed}] v 0) v1 -- 0x7f0ffc04bed0 con 0x7f0ffc0fcf80 2024-03-18T01:18:52.498 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:52.497+0000 7f0ff27fc700 1 -- 172.21.15.29:0/806820501 <== mon.1 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim_allowed}]=0 v21)=0 v21) v1 ==== 163+0+0 (secure 0 0 0) 0x7f0ffc04bed0 con 0x7f0ffc0fcf80 2024-03-18T01:18:52.499 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:52.499+0000 7f1003f95700 1 -- 172.21.15.29:0/806820501 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f0fe80692a0 msgr2=0x7f0fe806b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:18:52.500 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:52.499+0000 7f1003f95700 1 --2- 172.21.15.29:0/806820501 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f0fe80692a0 0x7f0fe806b750 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f0ff8004bc0 tx=0x7f0ff800b040 comp rx=0 tx=0).stop 2024-03-18T01:18:52.500 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:52.499+0000 7f1003f95700 1 -- 172.21.15.29:0/806820501 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0ffc0fcf80 msgr2=0x7f0ffc10ce00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:18:52.500 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:52.499+0000 7f1003f95700 1 --2- 172.21.15.29:0/806820501 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0ffc0fcf80 0x7f0ffc10ce00 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f0fec000c00 tx=0x7f0fec009e30 comp rx=0 tx=0).stop 2024-03-18T01:18:52.500 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:52.499+0000 7f1003f95700 1 -- 172.21.15.29:0/806820501 shutdown_connections 2024-03-18T01:18:52.500 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:52.499+0000 7f1003f95700 1 --2- 172.21.15.29:0/806820501 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f0fe80692a0 0x7f0fe806b750 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:18:52.500 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:52.499+0000 7f1003f95700 1 --2- 172.21.15.29:0/806820501 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0ffc0fbcd0 0x7f0ffc10c8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:18:52.500 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:52.499+0000 7f1003f95700 1 --2- 172.21.15.29:0/806820501 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0ffc0fcf80 0x7f0ffc10ce00 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:18:52.500 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:52.499+0000 7f1003f95700 1 -- 172.21.15.29:0/806820501 >> 172.21.15.29:0/806820501 conn(0x7f0ffc0f7430 msgr2=0x7f0ffc0f8a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:18:52.500 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:52.499+0000 7f1003f95700 1 -- 172.21.15.29:0/806820501 shutdown_connections 2024-03-18T01:18:52.500 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:52.499+0000 7f1003f95700 1 -- 172.21.15.29:0/806820501 wait complete. 2024-03-18T01:18:53.177 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 37f51182-e4c4-11ee-95c9-87774f69a715 -e sha1=3d78f5ba07bf18880172c07d4d7b4ac00c9d30af -- bash -c 'ceph config set global log_to_journald false --force' 2024-03-18T01:18:53.447 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/37f51182-e4c4-11ee-95c9-87774f69a715/mon.smithi029/config 2024-03-18T01:18:54.319 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:18:54 smithi029 ceph-mon[29331]: pgmap v205: 65 pgs: 65 active+clean; 586 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.7 KiB/s rd, 853 B/s wr, 5 op/s 2024-03-18T01:18:54.525 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:18:54 smithi090 ceph-mon[36989]: pgmap v205: 65 pgs: 65 active+clean; 586 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.7 KiB/s rd, 853 B/s wr, 5 op/s 2024-03-18T01:18:55.384 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:55.380+0000 7f56e823a700 1 -- 172.21.15.29:0/3551101483 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f56e00fe060 msgr2=0x7f56e00fe4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:18:55.385 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:55.380+0000 7f56e823a700 1 --2- 172.21.15.29:0/3551101483 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f56e00fe060 0x7f56e00fe4d0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f56d0004940 tx=0x7f56d0009e30 comp rx=0 tx=0).stop 2024-03-18T01:18:55.385 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:55.380+0000 7f56e823a700 1 -- 172.21.15.29:0/3551101483 shutdown_connections 2024-03-18T01:18:55.385 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:55.380+0000 7f56e823a700 1 --2- 172.21.15.29:0/3551101483 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f56e00fe060 0x7f56e00fe4d0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:18:55.385 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:55.380+0000 7f56e823a700 1 --2- 172.21.15.29:0/3551101483 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f56e00fcef0 0x7f56e00fd300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:18:55.385 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:55.380+0000 7f56e823a700 1 -- 172.21.15.29:0/3551101483 >> 172.21.15.29:0/3551101483 conn(0x7f56e00f85b0 msgr2=0x7f56e00fa9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:18:55.385 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:55.380+0000 7f56e823a700 1 -- 172.21.15.29:0/3551101483 shutdown_connections 2024-03-18T01:18:55.386 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:55.380+0000 7f56e823a700 1 -- 172.21.15.29:0/3551101483 wait complete. 2024-03-18T01:18:55.386 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:55.380+0000 7f56e823a700 1 Processor -- start 2024-03-18T01:18:55.388 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:55.381+0000 7f56e823a700 1 -- start start 2024-03-18T01:18:55.388 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:55.381+0000 7f56e823a700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f56e00fcef0 0x7f56e0071e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:18:55.388 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:55.381+0000 7f56e823a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f56e00fe060 0x7f56e0070310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:18:55.388 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:55.381+0000 7f56e823a700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56e00708c0 con 0x7f56e00fe060 2024-03-18T01:18:55.388 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:55.381+0000 7f56e823a700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56e0070a30 con 0x7f56e00fcef0 2024-03-18T01:18:55.388 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:55.381+0000 7f56e57d5700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f56e00fe060 0x7f56e0070310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:18:55.389 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:55.381+0000 7f56e57d5700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f56e00fe060 0x7f56e0070310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54878/0 (socket says 172.21.15.29:54878) 2024-03-18T01:18:55.389 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:55.381+0000 7f56e57d5700 1 -- 172.21.15.29:0/1879897473 learned_addr learned my addr 172.21.15.29:0/1879897473 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-18T01:18:55.389 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:55.381+0000 7f56e5fd6700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f56e00fcef0 0x7f56e0071e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:18:55.389 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:55.381+0000 7f56e57d5700 1 -- 172.21.15.29:0/1879897473 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f56e00fcef0 msgr2=0x7f56e0071e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:18:55.389 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:55.381+0000 7f56e57d5700 1 --2- 172.21.15.29:0/1879897473 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f56e00fcef0 0x7f56e0071e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:18:55.389 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:55.381+0000 7f56e57d5700 1 -- 172.21.15.29:0/1879897473 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56d0005040 con 0x7f56e00fe060 2024-03-18T01:18:55.389 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:55.381+0000 7f56e57d5700 1 --2- 172.21.15.29:0/1879897473 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f56e00fe060 0x7f56e0070310 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f56d0013870 tx=0x7f56d0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:18:55.389 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:55.382+0000 7f56d67fc700 1 -- 172.21.15.29:0/1879897473 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f56d0021070 con 0x7f56e00fe060 2024-03-18T01:18:55.389 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:55.382+0000 7f56e823a700 1 -- 172.21.15.29:0/1879897473 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56e0070cb0 con 0x7f56e00fe060 2024-03-18T01:18:55.389 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:55.382+0000 7f56d67fc700 1 -- 172.21.15.29:0/1879897473 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f56d0009430 con 0x7f56e00fe060 2024-03-18T01:18:55.389 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:55.382+0000 7f56e823a700 1 -- 172.21.15.29:0/1879897473 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56e01159f0 con 0x7f56e00fe060 2024-03-18T01:18:55.389 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:55.382+0000 7f56d67fc700 1 -- 172.21.15.29:0/1879897473 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f56d000e070 con 0x7f56e00fe060 2024-03-18T01:18:55.389 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:55.383+0000 7f56e823a700 1 -- 172.21.15.29:0/1879897473 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56e005f5f0 con 0x7f56e00fe060 2024-03-18T01:18:55.389 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:55.383+0000 7f56d67fc700 1 -- 172.21.15.29:0/1879897473 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f56d0013af0 con 0x7f56e00fe060 2024-03-18T01:18:55.390 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:55.383+0000 7f56d67fc700 1 --2- 172.21.15.29:0/1879897473 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f56cc0692a0 0x7f56cc06b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:18:55.390 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:55.384+0000 7f56d67fc700 1 -- 172.21.15.29:0/1879897473 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f56d00883e0 con 0x7f56e00fe060 2024-03-18T01:18:55.390 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:55.384+0000 7f56e5fd6700 1 --2- 172.21.15.29:0/1879897473 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f56cc0692a0 0x7f56cc06b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:18:55.390 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:55.384+0000 7f56e5fd6700 1 --2- 172.21.15.29:0/1879897473 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f56cc0692a0 0x7f56cc06b750 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f56e0071bd0 tx=0x7f56dc00b040 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:18:55.390 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:55.388+0000 7f56d67fc700 1 -- 172.21.15.29:0/1879897473 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175520 (secure 0 0 0) 0x7f56d0054760 con 0x7f56e00fe060 2024-03-18T01:18:55.590 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:55.588+0000 7f56e823a700 1 -- 172.21.15.29:0/1879897473 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command([{prefix=config set, name=log_to_journald}] v 0) v1 -- 0x7f56e004bed0 con 0x7f56e00fe060 2024-03-18T01:18:55.590 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:55.588+0000 7f56d67fc700 1 -- 172.21.15.29:0/1879897473 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{prefix=config set, name=log_to_journald}]=0 v21)=0 v21) v1 ==== 135+0+0 (secure 0 0 0) 0x7f56d0057d80 con 0x7f56e00fe060 2024-03-18T01:18:55.592 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:55.590+0000 7f56e823a700 1 -- 172.21.15.29:0/1879897473 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f56cc0692a0 msgr2=0x7f56cc06b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:18:55.592 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:55.590+0000 7f56e823a700 1 --2- 172.21.15.29:0/1879897473 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f56cc0692a0 0x7f56cc06b750 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f56e0071bd0 tx=0x7f56dc00b040 comp rx=0 tx=0).stop 2024-03-18T01:18:55.592 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:55.590+0000 7f56e823a700 1 -- 172.21.15.29:0/1879897473 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f56e00fe060 msgr2=0x7f56e0070310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:18:55.592 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:55.590+0000 7f56e823a700 1 --2- 172.21.15.29:0/1879897473 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f56e00fe060 0x7f56e0070310 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f56d0013870 tx=0x7f56d0009e30 comp rx=0 tx=0).stop 2024-03-18T01:18:55.592 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:55.590+0000 7f56e823a700 1 -- 172.21.15.29:0/1879897473 shutdown_connections 2024-03-18T01:18:55.592 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:55.590+0000 7f56e823a700 1 --2- 172.21.15.29:0/1879897473 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f56cc0692a0 0x7f56cc06b750 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:18:55.592 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:55.590+0000 7f56e823a700 1 --2- 172.21.15.29:0/1879897473 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f56e00fcef0 0x7f56e0071e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:18:55.592 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:55.590+0000 7f56e823a700 1 --2- 172.21.15.29:0/1879897473 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f56e00fe060 0x7f56e0070310 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:18:55.592 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:55.590+0000 7f56e823a700 1 -- 172.21.15.29:0/1879897473 >> 172.21.15.29:0/1879897473 conn(0x7f56e00f85b0 msgr2=0x7f56e00f9c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:18:55.592 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:55.591+0000 7f56e823a700 1 -- 172.21.15.29:0/1879897473 shutdown_connections 2024-03-18T01:18:55.592 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:55.591+0000 7f56e823a700 1 -- 172.21.15.29:0/1879897473 wait complete. 2024-03-18T01:18:56.319 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:18:56 smithi029 ceph-mon[29331]: pgmap v206: 65 pgs: 65 active+clean; 586 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.7 KiB/s rd, 853 B/s wr, 5 op/s 2024-03-18T01:18:56.319 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:18:56 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-18T01:18:56.379 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 37f51182-e4c4-11ee-95c9-87774f69a715 -e sha1=3d78f5ba07bf18880172c07d4d7b4ac00c9d30af -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1' 2024-03-18T01:18:56.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:18:56 smithi090 ceph-mon[36989]: pgmap v206: 65 pgs: 65 active+clean; 586 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.7 KiB/s rd, 853 B/s wr, 5 op/s 2024-03-18T01:18:56.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:18:56 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-18T01:18:56.657 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/37f51182-e4c4-11ee-95c9-87774f69a715/mon.smithi029/config 2024-03-18T01:18:58.257 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:18:58 smithi029 ceph-mon[29331]: pgmap v207: 65 pgs: 65 active+clean; 586 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.2 KiB/s rd, 341 B/s wr, 4 op/s 2024-03-18T01:18:58.476 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:58.473+0000 7f7a8a8eb700 1 -- 172.21.15.29:0/2480975708 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7a840fff70 msgr2=0x7f7a84102350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:18:58.476 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:58.473+0000 7f7a8a8eb700 1 --2- 172.21.15.29:0/2480975708 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7a840fff70 0x7f7a84102350 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f7a78004940 tx=0x7f7a78009e30 comp rx=0 tx=0).stop 2024-03-18T01:18:58.476 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:58.473+0000 7f7a8a8eb700 1 -- 172.21.15.29:0/2480975708 shutdown_connections 2024-03-18T01:18:58.476 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:58.473+0000 7f7a8a8eb700 1 --2- 172.21.15.29:0/2480975708 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7a840fff70 0x7f7a84102350 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:18:58.476 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:58.473+0000 7f7a8a8eb700 1 --2- 172.21.15.29:0/2480975708 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7a84066b10 0x7f7a840ffa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:18:58.476 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:58.473+0000 7f7a8a8eb700 1 -- 172.21.15.29:0/2480975708 >> 172.21.15.29:0/2480975708 conn(0x7f7a840f7440 msgr2=0x7f7a840f9870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:18:58.476 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:58.473+0000 7f7a8a8eb700 1 -- 172.21.15.29:0/2480975708 shutdown_connections 2024-03-18T01:18:58.477 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:58.473+0000 7f7a8a8eb700 1 -- 172.21.15.29:0/2480975708 wait complete. 2024-03-18T01:18:58.477 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:58.473+0000 7f7a8a8eb700 1 Processor -- start 2024-03-18T01:18:58.477 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:58.474+0000 7f7a8a8eb700 1 -- start start 2024-03-18T01:18:58.477 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:58.474+0000 7f7a8a8eb700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7a84066b10 0x7f7a8410da20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:18:58.477 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:58.474+0000 7f7a8a8eb700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7a840fff70 0x7f7a8410df60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:18:58.477 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:58.474+0000 7f7a8a8eb700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a8410e560 con 0x7f7a840fff70 2024-03-18T01:18:58.477 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:58.474+0000 7f7a8a8eb700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a8410ed60 con 0x7f7a84066b10 2024-03-18T01:18:58.477 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:58.474+0000 7f7a837fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7a840fff70 0x7f7a8410df60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:18:58.477 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:58.474+0000 7f7a837fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7a840fff70 0x7f7a8410df60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54900/0 (socket says 172.21.15.29:54900) 2024-03-18T01:18:58.477 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:58.474+0000 7f7a837fe700 1 -- 172.21.15.29:0/1862535852 learned_addr learned my addr 172.21.15.29:0/1862535852 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-18T01:18:58.477 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:58.474+0000 7f7a83fff700 1 --2- 172.21.15.29:0/1862535852 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7a84066b10 0x7f7a8410da20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:18:58.477 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:58.474+0000 7f7a837fe700 1 -- 172.21.15.29:0/1862535852 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7a84066b10 msgr2=0x7f7a8410da20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:18:58.477 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:58.474+0000 7f7a837fe700 1 --2- 172.21.15.29:0/1862535852 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7a84066b10 0x7f7a8410da20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:18:58.477 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:58.474+0000 7f7a837fe700 1 -- 172.21.15.29:0/1862535852 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a78005040 con 0x7f7a840fff70 2024-03-18T01:18:58.477 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:58.474+0000 7f7a837fe700 1 --2- 172.21.15.29:0/1862535852 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7a840fff70 0x7f7a8410df60 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f7a78013870 tx=0x7f7a78009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:18:58.477 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:58.474+0000 7f7a80ff9700 1 -- 172.21.15.29:0/1862535852 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7a78021070 con 0x7f7a840fff70 2024-03-18T01:18:58.478 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:58.475+0000 7f7a8a8eb700 1 -- 172.21.15.29:0/1862535852 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a8410efe0 con 0x7f7a840fff70 2024-03-18T01:18:58.478 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:58.475+0000 7f7a80ff9700 1 -- 172.21.15.29:0/1862535852 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7a78009430 con 0x7f7a840fff70 2024-03-18T01:18:58.478 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:58.475+0000 7f7a80ff9700 1 -- 172.21.15.29:0/1862535852 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7a7800e070 con 0x7f7a840fff70 2024-03-18T01:18:58.478 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:58.475+0000 7f7a8a8eb700 1 -- 172.21.15.29:0/1862535852 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a8410f520 con 0x7f7a840fff70 2024-03-18T01:18:58.479 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:58.476+0000 7f7a8a8eb700 1 -- 172.21.15.29:0/1862535852 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a8405f5f0 con 0x7f7a840fff70 2024-03-18T01:18:58.479 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:58.476+0000 7f7a80ff9700 1 -- 172.21.15.29:0/1862535852 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f7a7801c750 con 0x7f7a840fff70 2024-03-18T01:18:58.479 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:58.477+0000 7f7a80ff9700 1 --2- 172.21.15.29:0/1862535852 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f7a6c0692a0 0x7f7a6c06b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:18:58.479 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:58.477+0000 7f7a80ff9700 1 -- 172.21.15.29:0/1862535852 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f7a78088460 con 0x7f7a840fff70 2024-03-18T01:18:58.479 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:58.477+0000 7f7a83fff700 1 --2- 172.21.15.29:0/1862535852 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f7a6c0692a0 0x7f7a6c06b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:18:58.479 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:58.477+0000 7f7a83fff700 1 --2- 172.21.15.29:0/1862535852 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f7a6c0692a0 0x7f7a6c06b750 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f7a74009b80 tx=0x7f7a7400b040 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:18:58.482 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:58.481+0000 7f7a80ff9700 1 -- 172.21.15.29:0/1862535852 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175520 (secure 0 0 0) 0x7f7a780581f0 con 0x7f7a840fff70 2024-03-18T01:18:58.525 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:18:58 smithi090 ceph-mon[36989]: pgmap v207: 65 pgs: 65 active+clean; 586 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.2 KiB/s rd, 341 B/s wr, 4 op/s 2024-03-18T01:18:58.702 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:58.701+0000 7f7a8a8eb700 1 -- 172.21.15.29:0/1862535852 --> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] -- mgr_command(tid 0: {"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af", "target": ["mon-mgr", ""]}) v1 -- 0x7f7a8410f850 con 0x7f7a6c0692a0 2024-03-18T01:18:58.707 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:58.706+0000 7f7a80ff9700 1 -- 172.21.15.29:0/1862535852 <== mgr.14208 v2:172.21.15.29:6800/1641454714 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+89 (secure 0 0 0) 0x7f7a8410f850 con 0x7f7a6c0692a0 2024-03-18T01:18:58.708 INFO:teuthology.orchestra.run.smithi029.stdout:Initiating upgrade to quay.ceph.io/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af 2024-03-18T01:18:58.710 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:58.709+0000 7f7a8a8eb700 1 -- 172.21.15.29:0/1862535852 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f7a6c0692a0 msgr2=0x7f7a6c06b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:18:58.710 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:58.709+0000 7f7a8a8eb700 1 --2- 172.21.15.29:0/1862535852 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f7a6c0692a0 0x7f7a6c06b750 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f7a74009b80 tx=0x7f7a7400b040 comp rx=0 tx=0).stop 2024-03-18T01:18:58.711 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:58.709+0000 7f7a8a8eb700 1 -- 172.21.15.29:0/1862535852 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7a840fff70 msgr2=0x7f7a8410df60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:18:58.711 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:58.709+0000 7f7a8a8eb700 1 --2- 172.21.15.29:0/1862535852 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7a840fff70 0x7f7a8410df60 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f7a78013870 tx=0x7f7a78009e30 comp rx=0 tx=0).stop 2024-03-18T01:18:58.711 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:58.709+0000 7f7a8a8eb700 1 -- 172.21.15.29:0/1862535852 shutdown_connections 2024-03-18T01:18:58.711 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:58.709+0000 7f7a8a8eb700 1 --2- 172.21.15.29:0/1862535852 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f7a6c0692a0 0x7f7a6c06b750 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:18:58.711 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:58.709+0000 7f7a8a8eb700 1 --2- 172.21.15.29:0/1862535852 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7a84066b10 0x7f7a8410da20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:18:58.711 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:58.709+0000 7f7a8a8eb700 1 --2- 172.21.15.29:0/1862535852 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7a840fff70 0x7f7a8410df60 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:18:58.711 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:58.709+0000 7f7a8a8eb700 1 -- 172.21.15.29:0/1862535852 >> 172.21.15.29:0/1862535852 conn(0x7f7a840f7440 msgr2=0x7f7a840f7860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:18:58.711 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:58.709+0000 7f7a8a8eb700 1 -- 172.21.15.29:0/1862535852 shutdown_connections 2024-03-18T01:18:58.711 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:18:58.709+0000 7f7a8a8eb700 1 -- 172.21.15.29:0/1862535852 wait complete. 2024-03-18T01:18:59.465 INFO:teuthology.task.sequential:In sequential, running task cephadm.shell... 2024-03-18T01:18:59.466 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi029.front.sepia.ceph.com 2024-03-18T01:18:59.466 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 37f51182-e4c4-11ee-95c9-87774f69a715 -e sha1=3d78f5ba07bf18880172c07d4d7b4ac00c9d30af -- bash -c 'while ceph orch upgrade status | jq '"'"'.in_progress'"'"' | grep true && ! ceph orch upgrade status | jq '"'"'.message'"'"' | grep Error ; do ceph orch ps ; ceph versions ; ceph fs dump; ceph orch upgrade status ; ceph health detail ; sleep 30 ; done' 2024-03-18T01:18:59.716 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:18:59 smithi029 ceph-mon[29331]: from='client.14558 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T01:18:59.716 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:18:59 smithi029 ceph-mon[29331]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af 2024-03-18T01:18:59.716 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:18:59 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:18:59.716 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:18:59 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T01:18:59.716 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:18:59 smithi029 ceph-mon[29331]: pgmap v208: 65 pgs: 65 active+clean; 586 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 170 B/s wr, 4 op/s 2024-03-18T01:18:59.767 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/37f51182-e4c4-11ee-95c9-87774f69a715/mon.smithi029/config 2024-03-18T01:19:00.026 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:18:59 smithi090 ceph-mon[36989]: from='client.14558 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T01:19:00.026 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:18:59 smithi090 ceph-mon[36989]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af 2024-03-18T01:19:00.026 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:18:59 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:19:00.026 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:18:59 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T01:19:00.026 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:18:59 smithi090 ceph-mon[36989]: pgmap v208: 65 pgs: 65 active+clean; 586 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 170 B/s wr, 4 op/s 2024-03-18T01:19:01.026 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:19:00 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:19:01.026 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:19:00 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T01:19:01.026 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:19:00 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:19:01.026 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:19:00 smithi090 ceph-mon[36989]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af 2024-03-18T01:19:01.069 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:19:00 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:19:01.069 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:19:00 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T01:19:01.069 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:19:00 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:19:01.069 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:19:00 smithi029 ceph-mon[29331]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af 2024-03-18T01:19:01.753 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:19:01 smithi029 ceph-mon[29331]: pgmap v209: 65 pgs: 65 active+clean; 586 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 85 B/s wr, 4 op/s 2024-03-18T01:19:01.807 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:01.805+0000 7f288cb64700 1 -- 172.21.15.29:0/1616016232 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f28880fcf80 msgr2=0x7f28880fd3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:19:01.807 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:01.805+0000 7f288cb64700 1 --2- 172.21.15.29:0/1616016232 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f28880fcf80 0x7f28880fd3f0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f287c004940 tx=0x7f287c009e30 comp rx=0 tx=0).stop 2024-03-18T01:19:01.807 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:01.805+0000 7f288cb64700 1 -- 172.21.15.29:0/1616016232 shutdown_connections 2024-03-18T01:19:01.807 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:01.805+0000 7f288cb64700 1 --2- 172.21.15.29:0/1616016232 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f28880fcf80 0x7f28880fd3f0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:19:01.807 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:01.805+0000 7f288cb64700 1 --2- 172.21.15.29:0/1616016232 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f28880fbcd0 0x7f28880fc0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:19:01.807 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:01.805+0000 7f288cb64700 1 -- 172.21.15.29:0/1616016232 >> 172.21.15.29:0/1616016232 conn(0x7f28880f7430 msgr2=0x7f28880f9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:19:01.808 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:01.806+0000 7f288cb64700 1 -- 172.21.15.29:0/1616016232 shutdown_connections 2024-03-18T01:19:01.808 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:01.806+0000 7f288cb64700 1 -- 172.21.15.29:0/1616016232 wait complete. 2024-03-18T01:19:01.808 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:01.807+0000 7f288cb64700 1 Processor -- start 2024-03-18T01:19:01.808 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:01.807+0000 7f288cb64700 1 -- start start 2024-03-18T01:19:01.808 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:01.807+0000 7f288cb64700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f28880fbcd0 0x7f288810d830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:19:01.809 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:01.807+0000 7f288cb64700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f28880fcf80 0x7f288810dd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:19:01.809 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:01.807+0000 7f288cb64700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f288810e370 con 0x7f28880fbcd0 2024-03-18T01:19:01.809 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:01.807+0000 7f288cb64700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f288810eb70 con 0x7f28880fcf80 2024-03-18T01:19:01.809 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:01.807+0000 7f2885d9b700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f28880fcf80 0x7f288810dd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:19:01.809 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:01.807+0000 7f2885d9b700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f28880fcf80 0x7f288810dd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.29:59184/0 (socket says 172.21.15.29:59184) 2024-03-18T01:19:01.809 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:01.807+0000 7f2885d9b700 1 -- 172.21.15.29:0/2013397838 learned_addr learned my addr 172.21.15.29:0/2013397838 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-18T01:19:01.809 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:01.807+0000 7f288659c700 1 --2- 172.21.15.29:0/2013397838 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f28880fbcd0 0x7f288810d830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:19:01.810 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:01.808+0000 7f2885d9b700 1 -- 172.21.15.29:0/2013397838 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f28880fbcd0 msgr2=0x7f288810d830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:19:01.810 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:01.808+0000 7f2885d9b700 1 --2- 172.21.15.29:0/2013397838 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f28880fbcd0 0x7f288810d830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:19:01.810 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:01.808+0000 7f2885d9b700 1 -- 172.21.15.29:0/2013397838 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f287c005040 con 0x7f28880fcf80 2024-03-18T01:19:01.810 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:01.808+0000 7f288659c700 1 --2- 172.21.15.29:0/2013397838 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f28880fbcd0 0x7f288810d830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T01:19:01.810 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:01.808+0000 7f2885d9b700 1 --2- 172.21.15.29:0/2013397838 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f28880fcf80 0x7f288810dd70 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f287c000c00 tx=0x7f287c009e30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:19:01.810 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:01.808+0000 7f2876ffd700 1 -- 172.21.15.29:0/2013397838 <== mon.1 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f287c00e040 con 0x7f28880fcf80 2024-03-18T01:19:01.811 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:01.809+0000 7f2876ffd700 1 -- 172.21.15.29:0/2013397838 <== mon.1 v2:172.21.15.90:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f287c00be40 con 0x7f28880fcf80 2024-03-18T01:19:01.811 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:01.809+0000 7f2876ffd700 1 -- 172.21.15.29:0/2013397838 <== mon.1 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f287c0138a0 con 0x7f28880fcf80 2024-03-18T01:19:01.811 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:01.809+0000 7f288cb64700 1 -- 172.21.15.29:0/2013397838 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f288810edf0 con 0x7f28880fcf80 2024-03-18T01:19:01.811 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:01.809+0000 7f288cb64700 1 -- 172.21.15.29:0/2013397838 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f288810f270 con 0x7f28880fcf80 2024-03-18T01:19:01.811 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:01.810+0000 7f288cb64700 1 -- 172.21.15.29:0/2013397838 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f288805f5f0 con 0x7f28880fcf80 2024-03-18T01:19:01.812 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:01.811+0000 7f2876ffd700 1 -- 172.21.15.29:0/2013397838 <== mon.1 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f287c013b10 con 0x7f28880fcf80 2024-03-18T01:19:01.813 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:01.811+0000 7f2876ffd700 1 --2- 172.21.15.29:0/2013397838 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f28700692a0 0x7f287006b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:19:01.813 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:01.812+0000 7f288659c700 1 --2- 172.21.15.29:0/2013397838 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f28700692a0 0x7f287006b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:19:01.813 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:01.812+0000 7f2876ffd700 1 -- 172.21.15.29:0/2013397838 <== mon.1 v2:172.21.15.90:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f287c08c350 con 0x7f28880fcf80 2024-03-18T01:19:01.814 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:01.812+0000 7f288659c700 1 --2- 172.21.15.29:0/2013397838 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f28700692a0 0x7f287006b750 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f2878010520 tx=0x7f287800b040 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:19:01.821 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:01.820+0000 7f2876ffd700 1 -- 172.21.15.29:0/2013397838 <== mon.1 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175520 (secure 0 0 0) 0x7f287c05c0c0 con 0x7f28880fcf80 2024-03-18T01:19:02.026 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:19:01 smithi090 ceph-mon[36989]: pgmap v209: 65 pgs: 65 active+clean; 586 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 85 B/s wr, 4 op/s 2024-03-18T01:19:02.088 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:02.086+0000 7f288cb64700 1 -- 172.21.15.29:0/2013397838 --> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f288810f650 con 0x7f28700692a0 2024-03-18T01:19:02.089 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:02.088+0000 7f2876ffd700 1 -- 172.21.15.29:0/2013397838 <== mgr.14208 v2:172.21.15.29:6800/1641454714 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+358 (secure 0 0 0) 0x7f288810f650 con 0x7f28700692a0 2024-03-18T01:19:02.092 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:02.090+0000 7f288cb64700 1 -- 172.21.15.29:0/2013397838 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f28700692a0 msgr2=0x7f287006b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:19:02.092 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:02.090+0000 7f288cb64700 1 --2- 172.21.15.29:0/2013397838 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f28700692a0 0x7f287006b750 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f2878010520 tx=0x7f287800b040 comp rx=0 tx=0).stop 2024-03-18T01:19:02.092 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:02.090+0000 7f288cb64700 1 -- 172.21.15.29:0/2013397838 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f28880fcf80 msgr2=0x7f288810dd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:19:02.092 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:02.090+0000 7f288cb64700 1 --2- 172.21.15.29:0/2013397838 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f28880fcf80 0x7f288810dd70 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f287c000c00 tx=0x7f287c009e30 comp rx=0 tx=0).stop 2024-03-18T01:19:02.092 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:02.090+0000 7f288cb64700 1 -- 172.21.15.29:0/2013397838 shutdown_connections 2024-03-18T01:19:02.092 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:02.090+0000 7f288cb64700 1 --2- 172.21.15.29:0/2013397838 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f28700692a0 0x7f287006b750 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:19:02.092 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:02.090+0000 7f288cb64700 1 --2- 172.21.15.29:0/2013397838 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f28880fbcd0 0x7f288810d830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:19:02.092 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:02.090+0000 7f288cb64700 1 --2- 172.21.15.29:0/2013397838 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f28880fcf80 0x7f288810dd70 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:19:02.093 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:02.090+0000 7f288cb64700 1 -- 172.21.15.29:0/2013397838 >> 172.21.15.29:0/2013397838 conn(0x7f28880f7430 msgr2=0x7f28880f8790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:19:02.093 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:02.091+0000 7f288cb64700 1 -- 172.21.15.29:0/2013397838 shutdown_connections 2024-03-18T01:19:02.093 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:02.091+0000 7f288cb64700 1 -- 172.21.15.29:0/2013397838 wait complete. 2024-03-18T01:19:02.104 INFO:teuthology.orchestra.run.smithi029.stdout:true 2024-03-18T01:19:02.254 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:02.252+0000 7f38b8e50700 1 -- 172.21.15.29:0/967785146 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f38b40fb410 msgr2=0x7f38b4102690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:19:02.254 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:02.252+0000 7f38b8e50700 1 --2- 172.21.15.29:0/967785146 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f38b40fb410 0x7f38b4102690 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f38a8004940 tx=0x7f38a8009e30 comp rx=0 tx=0).stop 2024-03-18T01:19:02.254 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:02.252+0000 7f38b8e50700 1 -- 172.21.15.29:0/967785146 shutdown_connections 2024-03-18T01:19:02.254 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:02.252+0000 7f38b8e50700 1 --2- 172.21.15.29:0/967785146 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f38b40fb410 0x7f38b4102690 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:19:02.254 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:02.252+0000 7f38b8e50700 1 --2- 172.21.15.29:0/967785146 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f38b40faac0 0x7f38b40faed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:19:02.254 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:02.252+0000 7f38b8e50700 1 -- 172.21.15.29:0/967785146 >> 172.21.15.29:0/967785146 conn(0x7f38b40f6770 msgr2=0x7f38b40f8ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:19:02.254 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:02.252+0000 7f38b8e50700 1 -- 172.21.15.29:0/967785146 shutdown_connections 2024-03-18T01:19:02.254 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:02.252+0000 7f38b8e50700 1 -- 172.21.15.29:0/967785146 wait complete. 2024-03-18T01:19:02.256 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:02.253+0000 7f38b8e50700 1 Processor -- start 2024-03-18T01:19:02.256 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:02.253+0000 7f38b8e50700 1 -- start start 2024-03-18T01:19:02.256 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:02.253+0000 7f38b8e50700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f38b40faac0 0x7f38b40fded0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:19:02.256 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:02.253+0000 7f38b8e50700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f38b40fb410 0x7f38b40fc580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:19:02.256 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:02.253+0000 7f38b8e50700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38b40fe4d0 con 0x7f38b40faac0 2024-03-18T01:19:02.256 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:02.253+0000 7f38b8e50700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38b4111630 con 0x7f38b40fb410 2024-03-18T01:19:02.256 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:02.254+0000 7f38b1d9b700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f38b40fb410 0x7f38b40fc580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:19:02.256 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:02.254+0000 7f38b1d9b700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f38b40fb410 0x7f38b40fc580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.29:59208/0 (socket says 172.21.15.29:59208) 2024-03-18T01:19:02.257 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:02.254+0000 7f38b1d9b700 1 -- 172.21.15.29:0/3848465306 learned_addr learned my addr 172.21.15.29:0/3848465306 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-18T01:19:02.257 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:02.254+0000 7f38b259c700 1 --2- 172.21.15.29:0/3848465306 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f38b40faac0 0x7f38b40fded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:19:02.257 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:02.254+0000 7f38b1d9b700 1 -- 172.21.15.29:0/3848465306 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f38b40faac0 msgr2=0x7f38b40fded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:19:02.257 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:02.254+0000 7f38b1d9b700 1 --2- 172.21.15.29:0/3848465306 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f38b40faac0 0x7f38b40fded0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:19:02.257 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:02.254+0000 7f38b1d9b700 1 -- 172.21.15.29:0/3848465306 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38a8005040 con 0x7f38b40fb410 2024-03-18T01:19:02.257 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:02.254+0000 7f38b259c700 1 --2- 172.21.15.29:0/3848465306 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f38b40faac0 0x7f38b40fded0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T01:19:02.257 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:02.255+0000 7f38b1d9b700 1 --2- 172.21.15.29:0/3848465306 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f38b40fb410 0x7f38b40fc580 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f38a800b0d0 tx=0x7f38a8009e30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:19:02.257 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:02.255+0000 7f38a2ffd700 1 -- 172.21.15.29:0/3848465306 <== mon.1 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f38a800e030 con 0x7f38b40fb410 2024-03-18T01:19:02.257 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:02.255+0000 7f38a2ffd700 1 -- 172.21.15.29:0/3848465306 <== mon.1 v2:172.21.15.90:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f38a801c400 con 0x7f38b40fb410 2024-03-18T01:19:02.257 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:02.255+0000 7f38b8e50700 1 -- 172.21.15.29:0/3848465306 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38b4111880 con 0x7f38b40fb410 2024-03-18T01:19:02.258 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:02.255+0000 7f38a2ffd700 1 -- 172.21.15.29:0/3848465306 <== mon.1 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f38a8012600 con 0x7f38b40fb410 2024-03-18T01:19:02.258 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:02.255+0000 7f38b8e50700 1 -- 172.21.15.29:0/3848465306 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38b40fce30 con 0x7f38b40fb410 2024-03-18T01:19:02.258 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:02.257+0000 7f38b8e50700 1 -- 172.21.15.29:0/3848465306 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38b404c710 con 0x7f38b40fb410 2024-03-18T01:19:02.258 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:02.257+0000 7f38a2ffd700 1 -- 172.21.15.29:0/3848465306 <== mon.1 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f38a801c570 con 0x7f38b40fb410 2024-03-18T01:19:02.259 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:02.257+0000 7f38a2ffd700 1 --2- 172.21.15.29:0/3848465306 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f389c0692a0 0x7f389c06b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:19:02.259 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:02.257+0000 7f38a2ffd700 1 -- 172.21.15.29:0/3848465306 <== mon.1 v2:172.21.15.90:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f38a80884a0 con 0x7f38b40fb410 2024-03-18T01:19:02.259 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:02.258+0000 7f38b259c700 1 --2- 172.21.15.29:0/3848465306 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f389c0692a0 0x7f389c06b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:19:02.260 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:02.259+0000 7f38b259c700 1 --2- 172.21.15.29:0/3848465306 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f389c0692a0 0x7f389c06b750 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f38a4007fd0 tx=0x7f38a400b040 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:19:02.267 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:02.266+0000 7f38a2ffd700 1 -- 172.21.15.29:0/3848465306 <== mon.1 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175520 (secure 0 0 0) 0x7f38a8058230 con 0x7f38b40fb410 2024-03-18T01:19:02.686 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:02.684+0000 7f38b8e50700 1 -- 172.21.15.29:0/3848465306 --> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f38b41117c0 con 0x7f389c0692a0 2024-03-18T01:19:02.687 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:02.686+0000 7f38a2ffd700 1 -- 172.21.15.29:0/3848465306 <== mgr.14208 v2:172.21.15.29:6800/1641454714 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+358 (secure 0 0 0) 0x7f38b41117c0 con 0x7f389c0692a0 2024-03-18T01:19:02.690 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:02.689+0000 7f38b8e50700 1 -- 172.21.15.29:0/3848465306 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f389c0692a0 msgr2=0x7f389c06b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:19:02.690 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:02.689+0000 7f38b8e50700 1 --2- 172.21.15.29:0/3848465306 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f389c0692a0 0x7f389c06b750 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f38a4007fd0 tx=0x7f38a400b040 comp rx=0 tx=0).stop 2024-03-18T01:19:02.690 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:02.689+0000 7f38b8e50700 1 -- 172.21.15.29:0/3848465306 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f38b40fb410 msgr2=0x7f38b40fc580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:19:02.691 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:02.689+0000 7f38b8e50700 1 --2- 172.21.15.29:0/3848465306 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f38b40fb410 0x7f38b40fc580 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f38a800b0d0 tx=0x7f38a8009e30 comp rx=0 tx=0).stop 2024-03-18T01:19:02.691 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:02.689+0000 7f38b8e50700 1 -- 172.21.15.29:0/3848465306 shutdown_connections 2024-03-18T01:19:02.691 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:02.689+0000 7f38b8e50700 1 --2- 172.21.15.29:0/3848465306 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f389c0692a0 0x7f389c06b750 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:19:02.691 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:02.689+0000 7f38b8e50700 1 --2- 172.21.15.29:0/3848465306 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f38b40faac0 0x7f38b40fded0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:19:02.691 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:02.689+0000 7f38b8e50700 1 --2- 172.21.15.29:0/3848465306 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f38b40fb410 0x7f38b40fc580 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:19:02.691 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:02.689+0000 7f38b8e50700 1 -- 172.21.15.29:0/3848465306 >> 172.21.15.29:0/3848465306 conn(0x7f38b40f6770 msgr2=0x7f38b4100750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:19:02.691 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:02.689+0000 7f38b8e50700 1 -- 172.21.15.29:0/3848465306 shutdown_connections 2024-03-18T01:19:02.691 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:02.689+0000 7f38b8e50700 1 -- 172.21.15.29:0/3848465306 wait complete. 2024-03-18T01:19:02.864 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:02.862+0000 7f6f6836a700 1 -- 172.21.15.29:0/1811459744 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6f600fcf70 msgr2=0x7f6f600fd3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:19:02.864 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:02.862+0000 7f6f6836a700 1 --2- 172.21.15.29:0/1811459744 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6f600fcf70 0x7f6f600fd3e0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f6f50004940 tx=0x7f6f50009e30 comp rx=0 tx=0).stop 2024-03-18T01:19:02.864 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:02.862+0000 7f6f6836a700 1 -- 172.21.15.29:0/1811459744 shutdown_connections 2024-03-18T01:19:02.864 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:02.862+0000 7f6f6836a700 1 --2- 172.21.15.29:0/1811459744 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6f600fcf70 0x7f6f600fd3e0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:19:02.864 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:02.862+0000 7f6f6836a700 1 --2- 172.21.15.29:0/1811459744 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6f600fbe00 0x7f6f600fc210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:19:02.865 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:02.862+0000 7f6f6836a700 1 -- 172.21.15.29:0/1811459744 >> 172.21.15.29:0/1811459744 conn(0x7f6f600f74e0 msgr2=0x7f6f600f98f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:19:02.865 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:02.863+0000 7f6f6836a700 1 -- 172.21.15.29:0/1811459744 shutdown_connections 2024-03-18T01:19:02.865 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:02.863+0000 7f6f6836a700 1 -- 172.21.15.29:0/1811459744 wait complete. 2024-03-18T01:19:02.865 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:02.864+0000 7f6f6836a700 1 Processor -- start 2024-03-18T01:19:02.865 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:02.864+0000 7f6f6836a700 1 -- start start 2024-03-18T01:19:02.866 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:02.864+0000 7f6f6836a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6f600fbe00 0x7f6f6010c8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:19:02.866 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:02.865+0000 7f6f6836a700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6f600fcf70 0x7f6f6010cde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:19:02.867 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:02.865+0000 7f6f66106700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6f600fbe00 0x7f6f6010c8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:19:02.867 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:02.865+0000 7f6f66106700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6f600fbe00 0x7f6f6010c8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60476/0 (socket says 172.21.15.29:60476) 2024-03-18T01:19:02.867 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:02.865+0000 7f6f66106700 1 -- 172.21.15.29:0/4071486346 learned_addr learned my addr 172.21.15.29:0/4071486346 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-18T01:19:02.867 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:02.865+0000 7f6f65905700 1 --2- 172.21.15.29:0/4071486346 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6f600fcf70 0x7f6f6010cde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:19:02.867 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:02.865+0000 7f6f6836a700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f6010d3e0 con 0x7f6f600fbe00 2024-03-18T01:19:02.867 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:02.865+0000 7f6f6836a700 1 -- 172.21.15.29:0/4071486346 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f6010dc30 con 0x7f6f600fcf70 2024-03-18T01:19:02.868 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:02.866+0000 7f6f65905700 1 -- 172.21.15.29:0/4071486346 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6f600fbe00 msgr2=0x7f6f6010c8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:19:02.868 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:02.866+0000 7f6f65905700 1 --2- 172.21.15.29:0/4071486346 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6f600fbe00 0x7f6f6010c8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:19:02.868 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:02.866+0000 7f6f65905700 1 -- 172.21.15.29:0/4071486346 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f50005040 con 0x7f6f600fcf70 2024-03-18T01:19:02.868 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:02.866+0000 7f6f66106700 1 --2- 172.21.15.29:0/4071486346 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6f600fbe00 0x7f6f6010c8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T01:19:02.868 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:02.867+0000 7f6f65905700 1 --2- 172.21.15.29:0/4071486346 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6f600fcf70 0x7f6f6010cde0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f6f5000b0d0 tx=0x7f6f50009e30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:19:02.869 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:02.867+0000 7f6f56ffd700 1 -- 172.21.15.29:0/4071486346 <== mon.1 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6f5000e040 con 0x7f6f600fcf70 2024-03-18T01:19:02.869 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:02.867+0000 7f6f56ffd700 1 -- 172.21.15.29:0/4071486346 <== mon.1 v2:172.21.15.90:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6f5001e070 con 0x7f6f600fcf70 2024-03-18T01:19:02.869 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:02.867+0000 7f6f56ffd700 1 -- 172.21.15.29:0/4071486346 <== mon.1 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6f5000b650 con 0x7f6f600fcf70 2024-03-18T01:19:02.869 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:02.867+0000 7f6f6836a700 1 -- 172.21.15.29:0/4071486346 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f6010deb0 con 0x7f6f600fcf70 2024-03-18T01:19:02.869 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:02.867+0000 7f6f6836a700 1 -- 172.21.15.29:0/4071486346 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f6010e330 con 0x7f6f600fcf70 2024-03-18T01:19:02.870 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:02.869+0000 7f6f6836a700 1 -- 172.21.15.29:0/4071486346 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f6005f5f0 con 0x7f6f600fcf70 2024-03-18T01:19:02.871 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:02.870+0000 7f6f56ffd700 1 -- 172.21.15.29:0/4071486346 <== mon.1 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f6f5000b8c0 con 0x7f6f600fcf70 2024-03-18T01:19:02.871 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:02.870+0000 7f6f56ffd700 1 --2- 172.21.15.29:0/4071486346 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f6f4c0692b0 0x7f6f4c06b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:19:02.872 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:02.870+0000 7f6f56ffd700 1 -- 172.21.15.29:0/4071486346 <== mon.1 v2:172.21.15.90:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f6f5008a160 con 0x7f6f600fcf70 2024-03-18T01:19:02.872 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:02.871+0000 7f6f66106700 1 --2- 172.21.15.29:0/4071486346 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f6f4c0692b0 0x7f6f4c06b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:19:02.872 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:02.871+0000 7f6f66106700 1 --2- 172.21.15.29:0/4071486346 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f6f4c0692b0 0x7f6f4c06b760 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f6f5c0041d0 tx=0x7f6f5c00b040 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:19:02.880 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:02.879+0000 7f6f56ffd700 1 -- 172.21.15.29:0/4071486346 <== mon.1 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175520 (secure 0 0 0) 0x7f6f50059eb0 con 0x7f6f600fcf70 2024-03-18T01:19:03.026 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:19:02 smithi090 ceph-mon[36989]: from='client.24353 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T01:19:03.069 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:19:02 smithi029 ceph-mon[29331]: from='client.24353 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T01:19:03.285 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:03.283+0000 7f6f6836a700 1 -- 172.21.15.29:0/4071486346 --> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f6f6010e710 con 0x7f6f4c0692b0 2024-03-18T01:19:03.303 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:03.302+0000 7f6f56ffd700 1 -- 172.21.15.29:0/4071486346 <== mgr.14208 v2:172.21.15.29:6800/1641454714 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3366 (secure 0 0 0) 0x7f6f6010e710 con 0x7f6f4c0692b0 2024-03-18T01:19:03.303 INFO:teuthology.orchestra.run.smithi029.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-18T01:19:03.303 INFO:teuthology.orchestra.run.smithi029.stdout:alertmanager.smithi029 smithi029 *:9093,9094 running (5m) 31s ago 6m 24.9M - 0.25.0 c8568f914cd2 74aa94ac819f 2024-03-18T01:19:03.303 INFO:teuthology.orchestra.run.smithi029.stdout:crash.smithi029 smithi029 running (6m) 31s ago 6m 6660k - 17.2.7-684-g6716c9f3 19857db3290c a7a7a1f5abb1 2024-03-18T01:19:03.303 INFO:teuthology.orchestra.run.smithi029.stdout:crash.smithi090 smithi090 running (5m) 37s ago 5m 6665k - 17.2.7-684-g6716c9f3 19857db3290c a0efb58cf729 2024-03-18T01:19:03.304 INFO:teuthology.orchestra.run.smithi029.stdout:grafana.smithi029 smithi029 *:3000 running (4m) 31s ago 6m 86.6M - 9.4.7 954c08fa6188 875561d97b0e 2024-03-18T01:19:03.304 INFO:teuthology.orchestra.run.smithi029.stdout:mds.cephfs.smithi029.brxqeh smithi029 running (50s) 31s ago 50s 14.7M - 17.2.7-684-g6716c9f3 19857db3290c d569540253f3 2024-03-18T01:19:03.304 INFO:teuthology.orchestra.run.smithi029.stdout:mds.cephfs.smithi029.xuxvlr smithi029 running (56s) 31s ago 55s 11.3M - 17.2.7-684-g6716c9f3 19857db3290c 6d9efbc518c2 2024-03-18T01:19:03.304 INFO:teuthology.orchestra.run.smithi029.stdout:mds.cephfs.smithi090.bohder smithi090 running (54s) 37s ago 53s 14.4M - 17.2.7-684-g6716c9f3 19857db3290c 7726489bb8a0 2024-03-18T01:19:03.304 INFO:teuthology.orchestra.run.smithi029.stdout:mds.cephfs.smithi090.trsnxz smithi090 running (48s) 37s ago 47s 14.4M - 17.2.7-684-g6716c9f3 19857db3290c b4d306a4cb22 2024-03-18T01:19:03.304 INFO:teuthology.orchestra.run.smithi029.stdout:mgr.smithi029.qviwhl smithi029 *:9283 running (8m) 31s ago 8m 480M - 17.2.7-684-g6716c9f3 19857db3290c 3ff37450535a 2024-03-18T01:19:03.304 INFO:teuthology.orchestra.run.smithi029.stdout:mgr.smithi090.pnqedy smithi090 *:8443,9283 running (5m) 37s ago 5m 432M - 17.2.7-684-g6716c9f3 19857db3290c 68666c79e24d 2024-03-18T01:19:03.304 INFO:teuthology.orchestra.run.smithi029.stdout:mon.smithi029 smithi029 running (8m) 31s ago 8m 46.4M 2048M 17.2.7-684-g6716c9f3 19857db3290c 0a7fff1d6dfc 2024-03-18T01:19:03.304 INFO:teuthology.orchestra.run.smithi029.stdout:mon.smithi090 smithi090 running (5m) 37s ago 5m 38.8M 2048M 17.2.7-684-g6716c9f3 19857db3290c e23555eb8cd4 2024-03-18T01:19:03.304 INFO:teuthology.orchestra.run.smithi029.stdout:node-exporter.smithi029 smithi029 *:9100 running (6m) 31s ago 6m 15.8M - 1.5.0 0da6a335fe13 35b19f9a6014 2024-03-18T01:19:03.304 INFO:teuthology.orchestra.run.smithi029.stdout:node-exporter.smithi090 smithi090 *:9100 running (5m) 37s ago 5m 15.5M - 1.5.0 0da6a335fe13 46172a1dd8f1 2024-03-18T01:19:03.304 INFO:teuthology.orchestra.run.smithi029.stdout:osd.0 smithi029 running (4m) 31s ago 4m 46.2M 4289M 17.2.7-684-g6716c9f3 19857db3290c b34def6ff829 2024-03-18T01:19:03.304 INFO:teuthology.orchestra.run.smithi029.stdout:osd.1 smithi029 running (4m) 31s ago 4m 45.8M 4289M 17.2.7-684-g6716c9f3 19857db3290c f8915708098a 2024-03-18T01:19:03.304 INFO:teuthology.orchestra.run.smithi029.stdout:osd.2 smithi029 running (3m) 31s ago 3m 44.2M 4289M 17.2.7-684-g6716c9f3 19857db3290c cbf4647c043d 2024-03-18T01:19:03.305 INFO:teuthology.orchestra.run.smithi029.stdout:osd.3 smithi090 running (3m) 37s ago 3m 45.4M 5312M 17.2.7-684-g6716c9f3 19857db3290c dc4a91ea75c6 2024-03-18T01:19:03.305 INFO:teuthology.orchestra.run.smithi029.stdout:osd.4 smithi090 running (2m) 37s ago 2m 44.5M 5312M 17.2.7-684-g6716c9f3 19857db3290c 64754105e490 2024-03-18T01:19:03.305 INFO:teuthology.orchestra.run.smithi029.stdout:osd.5 smithi090 running (2m) 37s ago 2m 41.9M 5312M 17.2.7-684-g6716c9f3 19857db3290c 951ee2b3bc44 2024-03-18T01:19:03.305 INFO:teuthology.orchestra.run.smithi029.stdout:prometheus.smithi029 smithi029 *:9095 running (4m) 31s ago 6m 54.0M - 2.43.0 a07b618ecd1d a36de8869940 2024-03-18T01:19:03.306 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:03.304+0000 7f6f6836a700 1 -- 172.21.15.29:0/4071486346 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f6f4c0692b0 msgr2=0x7f6f4c06b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:19:03.306 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:03.304+0000 7f6f6836a700 1 --2- 172.21.15.29:0/4071486346 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f6f4c0692b0 0x7f6f4c06b760 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f6f5c0041d0 tx=0x7f6f5c00b040 comp rx=0 tx=0).stop 2024-03-18T01:19:03.306 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:03.304+0000 7f6f6836a700 1 -- 172.21.15.29:0/4071486346 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6f600fcf70 msgr2=0x7f6f6010cde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:19:03.306 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:03.304+0000 7f6f6836a700 1 --2- 172.21.15.29:0/4071486346 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6f600fcf70 0x7f6f6010cde0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f6f5000b0d0 tx=0x7f6f50009e30 comp rx=0 tx=0).stop 2024-03-18T01:19:03.306 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:03.305+0000 7f6f6836a700 1 -- 172.21.15.29:0/4071486346 shutdown_connections 2024-03-18T01:19:03.307 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:03.305+0000 7f6f6836a700 1 --2- 172.21.15.29:0/4071486346 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f6f4c0692b0 0x7f6f4c06b760 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:19:03.307 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:03.305+0000 7f6f6836a700 1 --2- 172.21.15.29:0/4071486346 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6f600fbe00 0x7f6f6010c8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:19:03.307 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:03.305+0000 7f6f6836a700 1 --2- 172.21.15.29:0/4071486346 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6f600fcf70 0x7f6f6010cde0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:19:03.307 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:03.305+0000 7f6f6836a700 1 -- 172.21.15.29:0/4071486346 >> 172.21.15.29:0/4071486346 conn(0x7f6f600f74e0 msgr2=0x7f6f600f8af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:19:03.307 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:03.305+0000 7f6f6836a700 1 -- 172.21.15.29:0/4071486346 shutdown_connections 2024-03-18T01:19:03.307 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:03.306+0000 7f6f6836a700 1 -- 172.21.15.29:0/4071486346 wait complete. 2024-03-18T01:19:03.461 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:03.459+0000 7fa8a9d9e700 1 -- 172.21.15.29:0/1984222286 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa8a40fcf80 msgr2=0x7fa8a40fd3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:19:03.461 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:03.459+0000 7fa8a9d9e700 1 --2- 172.21.15.29:0/1984222286 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa8a40fcf80 0x7fa8a40fd3f0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fa898004940 tx=0x7fa898009e30 comp rx=0 tx=0).stop 2024-03-18T01:19:03.461 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:03.459+0000 7fa8a9d9e700 1 -- 172.21.15.29:0/1984222286 shutdown_connections 2024-03-18T01:19:03.461 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:03.459+0000 7fa8a9d9e700 1 --2- 172.21.15.29:0/1984222286 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa8a40fcf80 0x7fa8a40fd3f0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:19:03.461 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:03.459+0000 7fa8a9d9e700 1 --2- 172.21.15.29:0/1984222286 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa8a40fbcd0 0x7fa8a40fc0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:19:03.461 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:03.459+0000 7fa8a9d9e700 1 -- 172.21.15.29:0/1984222286 >> 172.21.15.29:0/1984222286 conn(0x7fa8a40f7430 msgr2=0x7fa8a40f9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:19:03.461 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:03.459+0000 7fa8a9d9e700 1 -- 172.21.15.29:0/1984222286 shutdown_connections 2024-03-18T01:19:03.461 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:03.459+0000 7fa8a9d9e700 1 -- 172.21.15.29:0/1984222286 wait complete. 2024-03-18T01:19:03.463 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:03.460+0000 7fa8a9d9e700 1 Processor -- start 2024-03-18T01:19:03.463 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:03.460+0000 7fa8a9d9e700 1 -- start start 2024-03-18T01:19:03.463 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:03.460+0000 7fa8a9d9e700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa8a40fbcd0 0x7fa8a410d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:19:03.463 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:03.461+0000 7fa8a9d9e700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa8a40fcf80 0x7fa8a410df30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:19:03.463 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:03.461+0000 7fa8a9d9e700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8a410e530 con 0x7fa8a40fbcd0 2024-03-18T01:19:03.463 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:03.461+0000 7fa8a9d9e700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8a410ed30 con 0x7fa8a40fcf80 2024-03-18T01:19:03.463 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:03.461+0000 7fa8a37fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa8a40fbcd0 0x7fa8a410d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:19:03.463 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:03.461+0000 7fa8a37fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa8a40fbcd0 0x7fa8a410d9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60496/0 (socket says 172.21.15.29:60496) 2024-03-18T01:19:03.463 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:03.461+0000 7fa8a37fe700 1 -- 172.21.15.29:0/303054735 learned_addr learned my addr 172.21.15.29:0/303054735 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-18T01:19:03.463 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:03.461+0000 7fa8a37fe700 1 -- 172.21.15.29:0/303054735 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa8a40fcf80 msgr2=0x7fa8a410df30 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-18T01:19:03.464 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:03.461+0000 7fa8a37fe700 1 --2- 172.21.15.29:0/303054735 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa8a40fcf80 0x7fa8a410df30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:19:03.464 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:03.461+0000 7fa8a37fe700 1 -- 172.21.15.29:0/303054735 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa898005040 con 0x7fa8a40fbcd0 2024-03-18T01:19:03.464 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:03.461+0000 7fa8a37fe700 1 --2- 172.21.15.29:0/303054735 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa8a40fbcd0 0x7fa8a410d9f0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fa89400eb40 tx=0x7fa89400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:19:03.464 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:03.462+0000 7fa88bfff700 1 -- 172.21.15.29:0/303054735 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa894007cc0 con 0x7fa8a40fbcd0 2024-03-18T01:19:03.465 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:03.462+0000 7fa88bfff700 1 -- 172.21.15.29:0/303054735 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa89400f040 con 0x7fa8a40fbcd0 2024-03-18T01:19:03.465 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:03.462+0000 7fa88bfff700 1 -- 172.21.15.29:0/303054735 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa8940095e0 con 0x7fa8a40fbcd0 2024-03-18T01:19:03.465 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:03.462+0000 7fa8a9d9e700 1 -- 172.21.15.29:0/303054735 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8a410efb0 con 0x7fa8a40fbcd0 2024-03-18T01:19:03.465 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:03.462+0000 7fa8a9d9e700 1 -- 172.21.15.29:0/303054735 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8a410f430 con 0x7fa8a40fbcd0 2024-03-18T01:19:03.465 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:03.464+0000 7fa8a9d9e700 1 -- 172.21.15.29:0/303054735 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8a405f5f0 con 0x7fa8a40fbcd0 2024-03-18T01:19:03.466 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:03.464+0000 7fa88bfff700 1 -- 172.21.15.29:0/303054735 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7fa894009740 con 0x7fa8a40fbcd0 2024-03-18T01:19:03.466 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:03.464+0000 7fa88bfff700 1 --2- 172.21.15.29:0/303054735 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7fa88c0692a0 0x7fa88c06b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:19:03.466 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:03.464+0000 7fa88bfff700 1 -- 172.21.15.29:0/303054735 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7fa894086d60 con 0x7fa8a40fbcd0 2024-03-18T01:19:03.466 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:03.465+0000 7fa8a2ffd700 1 --2- 172.21.15.29:0/303054735 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7fa88c0692a0 0x7fa88c06b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:19:03.467 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:03.465+0000 7fa8a2ffd700 1 --2- 172.21.15.29:0/303054735 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7fa88c0692a0 0x7fa88c06b750 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fa898013870 tx=0x7fa898009e30 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:19:03.470 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:03.469+0000 7fa88bfff700 1 -- 172.21.15.29:0/303054735 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175520 (secure 0 0 0) 0x7fa89404ddb0 con 0x7fa8a40fbcd0 2024-03-18T01:19:03.943 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:03.942+0000 7fa8a9d9e700 1 -- 172.21.15.29:0/303054735 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fa8a410f7a0 con 0x7fa8a40fbcd0 2024-03-18T01:19:03.944 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:03.943+0000 7fa88bfff700 1 -- 172.21.15.29:0/303054735 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+638 (secure 0 0 0) 0x7fa894056b80 con 0x7fa8a40fbcd0 2024-03-18T01:19:03.945 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-03-18T01:19:03.945 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": { 2024-03-18T01:19:03.945 INFO:teuthology.orchestra.run.smithi029.stdout: "ceph version 17.2.7-684-g6716c9f3 (6716c9f3976cf9fc2fd546e9f1182c2d4421b451) quincy (stable)": 2 2024-03-18T01:19:03.945 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-18T01:19:03.945 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": { 2024-03-18T01:19:03.945 INFO:teuthology.orchestra.run.smithi029.stdout: "ceph version 17.2.7-684-g6716c9f3 (6716c9f3976cf9fc2fd546e9f1182c2d4421b451) quincy (stable)": 2 2024-03-18T01:19:03.946 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-18T01:19:03.946 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": { 2024-03-18T01:19:03.946 INFO:teuthology.orchestra.run.smithi029.stdout: "ceph version 17.2.7-684-g6716c9f3 (6716c9f3976cf9fc2fd546e9f1182c2d4421b451) quincy (stable)": 6 2024-03-18T01:19:03.946 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-18T01:19:03.946 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": { 2024-03-18T01:19:03.946 INFO:teuthology.orchestra.run.smithi029.stdout: "ceph version 17.2.7-684-g6716c9f3 (6716c9f3976cf9fc2fd546e9f1182c2d4421b451) quincy (stable)": 4 2024-03-18T01:19:03.946 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-18T01:19:03.947 INFO:teuthology.orchestra.run.smithi029.stdout: "overall": { 2024-03-18T01:19:03.947 INFO:teuthology.orchestra.run.smithi029.stdout: "ceph version 17.2.7-684-g6716c9f3 (6716c9f3976cf9fc2fd546e9f1182c2d4421b451) quincy (stable)": 14 2024-03-18T01:19:03.947 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-18T01:19:03.947 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-03-18T01:19:03.948 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:03.945+0000 7fa8a9d9e700 1 -- 172.21.15.29:0/303054735 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7fa88c0692a0 msgr2=0x7fa88c06b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:19:03.949 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:03.945+0000 7fa8a9d9e700 1 --2- 172.21.15.29:0/303054735 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7fa88c0692a0 0x7fa88c06b750 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fa898013870 tx=0x7fa898009e30 comp rx=0 tx=0).stop 2024-03-18T01:19:03.949 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:03.945+0000 7fa8a9d9e700 1 -- 172.21.15.29:0/303054735 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa8a40fbcd0 msgr2=0x7fa8a410d9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:19:03.949 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:03.945+0000 7fa8a9d9e700 1 --2- 172.21.15.29:0/303054735 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa8a40fbcd0 0x7fa8a410d9f0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fa89400eb40 tx=0x7fa89400b040 comp rx=0 tx=0).stop 2024-03-18T01:19:03.949 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:03.946+0000 7fa8a9d9e700 1 -- 172.21.15.29:0/303054735 shutdown_connections 2024-03-18T01:19:03.949 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:03.946+0000 7fa8a9d9e700 1 --2- 172.21.15.29:0/303054735 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7fa88c0692a0 0x7fa88c06b750 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:19:03.949 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:03.946+0000 7fa8a9d9e700 1 --2- 172.21.15.29:0/303054735 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa8a40fbcd0 0x7fa8a410d9f0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:19:03.949 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:03.946+0000 7fa8a9d9e700 1 --2- 172.21.15.29:0/303054735 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa8a40fcf80 0x7fa8a410df30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:19:03.950 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:03.946+0000 7fa8a9d9e700 1 -- 172.21.15.29:0/303054735 >> 172.21.15.29:0/303054735 conn(0x7fa8a40f7430 msgr2=0x7fa8a40f8ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:19:03.950 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:03.946+0000 7fa8a9d9e700 1 -- 172.21.15.29:0/303054735 shutdown_connections 2024-03-18T01:19:03.950 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:03.947+0000 7fa8a9d9e700 1 -- 172.21.15.29:0/303054735 wait complete. 2024-03-18T01:19:04.026 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:19:03 smithi090 ceph-mon[36989]: from='client.24357 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T01:19:04.026 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:19:03 smithi090 ceph-mon[36989]: pgmap v210: 65 pgs: 65 active+clean; 586 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 85 B/s wr, 4 op/s 2024-03-18T01:19:04.026 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:19:03 smithi090 ceph-mon[36989]: from='client.24361 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T01:19:04.069 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:19:03 smithi029 ceph-mon[29331]: from='client.24357 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T01:19:04.069 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:19:03 smithi029 ceph-mon[29331]: pgmap v210: 65 pgs: 65 active+clean; 586 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 85 B/s wr, 4 op/s 2024-03-18T01:19:04.069 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:19:03 smithi029 ceph-mon[29331]: from='client.24361 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T01:19:04.116 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:04.114+0000 7f2c278e9700 1 -- 172.21.15.29:0/2802264739 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2c200fce40 msgr2=0x7f2c200fd250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:19:04.117 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:04.115+0000 7f2c278e9700 1 --2- 172.21.15.29:0/2802264739 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2c200fce40 0x7f2c200fd250 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f2c1c004940 tx=0x7f2c1c009e30 comp rx=0 tx=0).stop 2024-03-18T01:19:04.117 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:04.115+0000 7f2c278e9700 1 -- 172.21.15.29:0/2802264739 shutdown_connections 2024-03-18T01:19:04.117 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:04.115+0000 7f2c278e9700 1 --2- 172.21.15.29:0/2802264739 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2c200fe040 0x7f2c200fe4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:19:04.117 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:04.115+0000 7f2c278e9700 1 --2- 172.21.15.29:0/2802264739 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2c200fce40 0x7f2c200fd250 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:19:04.117 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:04.115+0000 7f2c278e9700 1 -- 172.21.15.29:0/2802264739 >> 172.21.15.29:0/2802264739 conn(0x7f2c200f8550 msgr2=0x7f2c200fa980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:19:04.117 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:04.115+0000 7f2c278e9700 1 -- 172.21.15.29:0/2802264739 shutdown_connections 2024-03-18T01:19:04.117 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:04.115+0000 7f2c278e9700 1 -- 172.21.15.29:0/2802264739 wait complete. 2024-03-18T01:19:04.118 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:04.116+0000 7f2c278e9700 1 Processor -- start 2024-03-18T01:19:04.118 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:04.116+0000 7f2c278e9700 1 -- start start 2024-03-18T01:19:04.118 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:04.116+0000 7f2c278e9700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2c200fce40 0x7f2c20071df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:19:04.119 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:04.116+0000 7f2c278e9700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2c200fe040 0x7f2c20070310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:19:04.119 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:04.116+0000 7f2c25685700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2c200fce40 0x7f2c20071df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:19:04.119 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:04.116+0000 7f2c24e84700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2c200fe040 0x7f2c20070310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:19:04.119 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:04.116+0000 7f2c25685700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2c200fce40 0x7f2c20071df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60510/0 (socket says 172.21.15.29:60510) 2024-03-18T01:19:04.119 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:04.116+0000 7f2c25685700 1 -- 172.21.15.29:0/4124482494 learned_addr learned my addr 172.21.15.29:0/4124482494 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-18T01:19:04.119 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:04.116+0000 7f2c278e9700 1 -- 172.21.15.29:0/4124482494 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c20111490 con 0x7f2c200fce40 2024-03-18T01:19:04.119 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:04.116+0000 7f2c278e9700 1 -- 172.21.15.29:0/4124482494 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c201115d0 con 0x7f2c200fe040 2024-03-18T01:19:04.119 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:04.117+0000 7f2c25685700 1 -- 172.21.15.29:0/4124482494 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2c200fe040 msgr2=0x7f2c20070310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:19:04.119 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:04.117+0000 7f2c25685700 1 --2- 172.21.15.29:0/4124482494 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2c200fe040 0x7f2c20070310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:19:04.119 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:04.117+0000 7f2c25685700 1 -- 172.21.15.29:0/4124482494 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c1c005040 con 0x7f2c200fce40 2024-03-18T01:19:04.119 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:04.117+0000 7f2c25685700 1 --2- 172.21.15.29:0/4124482494 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2c200fce40 0x7f2c20071df0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f2c1c013870 tx=0x7f2c1c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:19:04.120 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:04.117+0000 7f2c15ffb700 1 -- 172.21.15.29:0/4124482494 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2c1c00e030 con 0x7f2c200fce40 2024-03-18T01:19:04.120 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:04.118+0000 7f2c278e9700 1 -- 172.21.15.29:0/4124482494 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c200708b0 con 0x7f2c200fce40 2024-03-18T01:19:04.120 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:04.118+0000 7f2c15ffb700 1 -- 172.21.15.29:0/4124482494 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2c1c01c8d0 con 0x7f2c200fce40 2024-03-18T01:19:04.120 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:04.118+0000 7f2c278e9700 1 -- 172.21.15.29:0/4124482494 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c20070df0 con 0x7f2c200fce40 2024-03-18T01:19:04.120 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:04.119+0000 7f2c15ffb700 1 -- 172.21.15.29:0/4124482494 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2c1c00b790 con 0x7f2c200fce40 2024-03-18T01:19:04.121 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:04.120+0000 7f2c0b7fe700 1 -- 172.21.15.29:0/4124482494 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c2005f5f0 con 0x7f2c200fce40 2024-03-18T01:19:04.122 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:04.121+0000 7f2c15ffb700 1 -- 172.21.15.29:0/4124482494 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f2c1c00b8f0 con 0x7f2c200fce40 2024-03-18T01:19:04.124 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:04.121+0000 7f2c15ffb700 1 --2- 172.21.15.29:0/4124482494 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f2c0c0692f0 0x7f2c0c06b7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:19:04.124 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:04.122+0000 7f2c15ffb700 1 -- 172.21.15.29:0/4124482494 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f2c1c089150 con 0x7f2c200fce40 2024-03-18T01:19:04.124 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:04.122+0000 7f2c24e84700 1 --2- 172.21.15.29:0/4124482494 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f2c0c0692f0 0x7f2c0c06b7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:19:04.124 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:04.123+0000 7f2c24e84700 1 --2- 172.21.15.29:0/4124482494 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f2c0c0692f0 0x7f2c0c06b7a0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f2c200723d0 tx=0x7f2c1000b040 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:19:04.128 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:04.127+0000 7f2c15ffb700 1 -- 172.21.15.29:0/4124482494 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175520 (secure 0 0 0) 0x7f2c1c055470 con 0x7f2c200fce40 2024-03-18T01:19:04.499 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:04.498+0000 7f2c0b7fe700 1 -- 172.21.15.29:0/4124482494 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f2c2004bed0 con 0x7f2c200fce40 2024-03-18T01:19:04.500 INFO:teuthology.orchestra.run.smithi029.stdout:e20 2024-03-18T01:19:04.500 INFO:teuthology.orchestra.run.smithi029.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-18T01:19:04.500 INFO:teuthology.orchestra.run.smithi029.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-18T01:19:04.500 INFO:teuthology.orchestra.run.smithi029.stdout:legacy client fscid: 1 2024-03-18T01:19:04.500 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-18T01:19:04.500 INFO:teuthology.orchestra.run.smithi029.stdout:Filesystem 'cephfs' (1) 2024-03-18T01:19:04.500 INFO:teuthology.orchestra.run.smithi029.stdout:fs_name cephfs 2024-03-18T01:19:04.501 INFO:teuthology.orchestra.run.smithi029.stdout:epoch 20 2024-03-18T01:19:04.501 INFO:teuthology.orchestra.run.smithi029.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-03-18T01:19:04.501 INFO:teuthology.orchestra.run.smithi029.stdout:created 2024-03-18T01:18:03.228199+0000 2024-03-18T01:19:04.501 INFO:teuthology.orchestra.run.smithi029.stdout:modified 2024-03-18T01:18:21.293804+0000 2024-03-18T01:19:04.501 INFO:teuthology.orchestra.run.smithi029.stdout:tableserver 0 2024-03-18T01:19:04.501 INFO:teuthology.orchestra.run.smithi029.stdout:root 0 2024-03-18T01:19:04.501 INFO:teuthology.orchestra.run.smithi029.stdout:session_timeout 60 2024-03-18T01:19:04.501 INFO:teuthology.orchestra.run.smithi029.stdout:session_autoclose 300 2024-03-18T01:19:04.501 INFO:teuthology.orchestra.run.smithi029.stdout:max_file_size 1099511627776 2024-03-18T01:19:04.501 INFO:teuthology.orchestra.run.smithi029.stdout:required_client_features {} 2024-03-18T01:19:04.501 INFO:teuthology.orchestra.run.smithi029.stdout:last_failure 0 2024-03-18T01:19:04.501 INFO:teuthology.orchestra.run.smithi029.stdout:last_failure_osd_epoch 46 2024-03-18T01:19:04.501 INFO:teuthology.orchestra.run.smithi029.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-18T01:19:04.501 INFO:teuthology.orchestra.run.smithi029.stdout:max_mds 2 2024-03-18T01:19:04.501 INFO:teuthology.orchestra.run.smithi029.stdout:in 0,1 2024-03-18T01:19:04.502 INFO:teuthology.orchestra.run.smithi029.stdout:up {0=14480,1=24281} 2024-03-18T01:19:04.502 INFO:teuthology.orchestra.run.smithi029.stdout:failed 2024-03-18T01:19:04.502 INFO:teuthology.orchestra.run.smithi029.stdout:damaged 2024-03-18T01:19:04.502 INFO:teuthology.orchestra.run.smithi029.stdout:stopped 2024-03-18T01:19:04.502 INFO:teuthology.orchestra.run.smithi029.stdout:data_pools [3] 2024-03-18T01:19:04.502 INFO:teuthology.orchestra.run.smithi029.stdout:metadata_pool 2 2024-03-18T01:19:04.502 INFO:teuthology.orchestra.run.smithi029.stdout:inline_data enabled 2024-03-18T01:19:04.502 INFO:teuthology.orchestra.run.smithi029.stdout:balancer 2024-03-18T01:19:04.502 INFO:teuthology.orchestra.run.smithi029.stdout:standby_count_wanted 1 2024-03-18T01:19:04.502 INFO:teuthology.orchestra.run.smithi029.stdout:[mds.cephfs.smithi029.brxqeh{0:14480} state up:active seq 6 join_fscid=1 addr [v2:172.21.15.29:6828/1299751620,v1:172.21.15.29:6829/1299751620] compat {c=[1],r=[1],i=[7ff]}] 2024-03-18T01:19:04.502 INFO:teuthology.orchestra.run.smithi029.stdout:[mds.cephfs.smithi029.xuxvlr{0:14492} state up:standby-replay seq 1 join_fscid=1 addr [v2:172.21.15.29:6826/890079928,v1:172.21.15.29:6827/890079928] compat {c=[1],r=[1],i=[7ff]}] 2024-03-18T01:19:04.502 INFO:teuthology.orchestra.run.smithi029.stdout:[mds.cephfs.smithi090.bohder{1:24281} state up:active seq 4 join_fscid=1 addr [v2:172.21.15.90:6824/1998449484,v1:172.21.15.90:6825/1998449484] compat {c=[1],r=[1],i=[7ff]}] 2024-03-18T01:19:04.502 INFO:teuthology.orchestra.run.smithi029.stdout:[mds.cephfs.smithi090.trsnxz{1:24297} state up:standby-replay seq 2 join_fscid=1 addr [v2:172.21.15.90:6826/439410941,v1:172.21.15.90:6827/439410941] compat {c=[1],r=[1],i=[7ff]}] 2024-03-18T01:19:04.502 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-18T01:19:04.502 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-18T01:19:04.503 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:04.499+0000 7f2c15ffb700 1 -- 172.21.15.29:0/4124482494 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 20 v20) v1 ==== 76+0+1825 (secure 0 0 0) 0x7f2c1c058a90 con 0x7f2c200fce40 2024-03-18T01:19:04.503 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:04.502+0000 7f2c0b7fe700 1 -- 172.21.15.29:0/4124482494 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f2c0c0692f0 msgr2=0x7f2c0c06b7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:19:04.503 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:04.502+0000 7f2c0b7fe700 1 --2- 172.21.15.29:0/4124482494 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f2c0c0692f0 0x7f2c0c06b7a0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f2c200723d0 tx=0x7f2c1000b040 comp rx=0 tx=0).stop 2024-03-18T01:19:04.504 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:04.502+0000 7f2c0b7fe700 1 -- 172.21.15.29:0/4124482494 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2c200fce40 msgr2=0x7f2c20071df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:19:04.504 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:04.502+0000 7f2c0b7fe700 1 --2- 172.21.15.29:0/4124482494 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2c200fce40 0x7f2c20071df0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f2c1c013870 tx=0x7f2c1c009e30 comp rx=0 tx=0).stop 2024-03-18T01:19:04.504 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:04.502+0000 7f2c0b7fe700 1 -- 172.21.15.29:0/4124482494 shutdown_connections 2024-03-18T01:19:04.505 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:04.502+0000 7f2c0b7fe700 1 --2- 172.21.15.29:0/4124482494 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f2c0c0692f0 0x7f2c0c06b7a0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:19:04.505 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:04.502+0000 7f2c0b7fe700 1 --2- 172.21.15.29:0/4124482494 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2c200fce40 0x7f2c20071df0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:19:04.505 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:04.502+0000 7f2c0b7fe700 1 --2- 172.21.15.29:0/4124482494 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2c200fe040 0x7f2c20070310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:19:04.505 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:04.502+0000 7f2c0b7fe700 1 -- 172.21.15.29:0/4124482494 >> 172.21.15.29:0/4124482494 conn(0x7f2c200f8550 msgr2=0x7f2c200f98b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:19:04.505 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:04.503+0000 7f2c0b7fe700 1 -- 172.21.15.29:0/4124482494 shutdown_connections 2024-03-18T01:19:04.505 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:04.503+0000 7f2c0b7fe700 1 -- 172.21.15.29:0/4124482494 wait complete. 2024-03-18T01:19:04.505 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 20 2024-03-18T01:19:04.663 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:04.660+0000 7f019c818700 1 -- 172.21.15.29:0/551154674 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f01980fbce0 msgr2=0x7f01980fc0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:19:04.663 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:04.660+0000 7f019c818700 1 --2- 172.21.15.29:0/551154674 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f01980fbce0 0x7f01980fc0f0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f0188004940 tx=0x7f0188009e30 comp rx=0 tx=0).stop 2024-03-18T01:19:04.663 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:04.661+0000 7f019c818700 1 -- 172.21.15.29:0/551154674 shutdown_connections 2024-03-18T01:19:04.663 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:04.661+0000 7f019c818700 1 --2- 172.21.15.29:0/551154674 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f01980fcf90 0x7f01980fd400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:19:04.663 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:04.661+0000 7f019c818700 1 --2- 172.21.15.29:0/551154674 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f01980fbce0 0x7f01980fc0f0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:19:04.663 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:04.661+0000 7f019c818700 1 -- 172.21.15.29:0/551154674 >> 172.21.15.29:0/551154674 conn(0x7f01980f7440 msgr2=0x7f01980f9870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:19:04.663 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:04.661+0000 7f019c818700 1 -- 172.21.15.29:0/551154674 shutdown_connections 2024-03-18T01:19:04.663 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:04.661+0000 7f019c818700 1 -- 172.21.15.29:0/551154674 wait complete. 2024-03-18T01:19:04.664 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:04.662+0000 7f019c818700 1 Processor -- start 2024-03-18T01:19:04.664 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:04.662+0000 7f019c818700 1 -- start start 2024-03-18T01:19:04.665 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:04.662+0000 7f019c818700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f01980fbce0 0x7f019810d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:19:04.665 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:04.662+0000 7f019c818700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f01980fcf90 0x7f019810df00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:19:04.665 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:04.662+0000 7f019c818700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f019810e500 con 0x7f01980fbce0 2024-03-18T01:19:04.665 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:04.662+0000 7f019c818700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f019810ed50 con 0x7f01980fcf90 2024-03-18T01:19:04.665 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:04.663+0000 7f0195d9b700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f01980fcf90 0x7f019810df00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:19:04.665 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:04.663+0000 7f0195d9b700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f01980fcf90 0x7f019810df00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.29:33804/0 (socket says 172.21.15.29:33804) 2024-03-18T01:19:04.665 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:04.663+0000 7f0195d9b700 1 -- 172.21.15.29:0/1107796317 learned_addr learned my addr 172.21.15.29:0/1107796317 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-18T01:19:04.665 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:04.663+0000 7f019659c700 1 --2- 172.21.15.29:0/1107796317 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f01980fbce0 0x7f019810d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:19:04.665 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:04.663+0000 7f019659c700 1 -- 172.21.15.29:0/1107796317 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f01980fcf90 msgr2=0x7f019810df00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:19:04.665 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:04.663+0000 7f019659c700 1 --2- 172.21.15.29:0/1107796317 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f01980fcf90 0x7f019810df00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:19:04.665 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:04.663+0000 7f019659c700 1 -- 172.21.15.29:0/1107796317 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0188005040 con 0x7f01980fbce0 2024-03-18T01:19:04.666 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:04.664+0000 7f019659c700 1 --2- 172.21.15.29:0/1107796317 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f01980fbce0 0x7f019810d9c0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f0188000c00 tx=0x7f0188009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:19:04.666 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:04.664+0000 7f0186ffd700 1 -- 172.21.15.29:0/1107796317 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f018800e070 con 0x7f01980fbce0 2024-03-18T01:19:04.666 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:04.664+0000 7f0186ffd700 1 -- 172.21.15.29:0/1107796317 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f018801c400 con 0x7f01980fbce0 2024-03-18T01:19:04.666 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:04.664+0000 7f0186ffd700 1 -- 172.21.15.29:0/1107796317 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f018800b670 con 0x7f01980fbce0 2024-03-18T01:19:04.666 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:04.664+0000 7f019c818700 1 -- 172.21.15.29:0/1107796317 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f019810ef70 con 0x7f01980fbce0 2024-03-18T01:19:04.666 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:04.664+0000 7f019c818700 1 -- 172.21.15.29:0/1107796317 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f019810f3f0 con 0x7f01980fbce0 2024-03-18T01:19:04.667 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:04.666+0000 7f019c818700 1 -- 172.21.15.29:0/1107796317 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f019805f5f0 con 0x7f01980fbce0 2024-03-18T01:19:04.668 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:04.667+0000 7f0186ffd700 1 -- 172.21.15.29:0/1107796317 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f018801c9e0 con 0x7f01980fbce0 2024-03-18T01:19:04.668 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:04.667+0000 7f0186ffd700 1 --2- 172.21.15.29:0/1107796317 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f01800692a0 0x7f018006b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:19:04.668 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:04.667+0000 7f0186ffd700 1 -- 172.21.15.29:0/1107796317 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f0188088550 con 0x7f01980fbce0 2024-03-18T01:19:04.668 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:04.667+0000 7f0195d9b700 1 --2- 172.21.15.29:0/1107796317 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f01800692a0 0x7f018006b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:19:04.669 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:04.668+0000 7f0195d9b700 1 --2- 172.21.15.29:0/1107796317 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f01800692a0 0x7f018006b750 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f018c008fd0 tx=0x7f018c00b040 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:19:04.677 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:04.675+0000 7f0186ffd700 1 -- 172.21.15.29:0/1107796317 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175520 (secure 0 0 0) 0x7f0188058360 con 0x7f01980fbce0 2024-03-18T01:19:04.819 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:19:04 smithi029 ceph-mon[29331]: from='client.? 172.21.15.29:0/303054735' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-18T01:19:04.819 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:19:04 smithi029 ceph-mon[29331]: from='client.? 172.21.15.29:0/4124482494' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-18T01:19:05.026 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:19:04 smithi090 ceph-mon[36989]: from='client.? 172.21.15.29:0/303054735' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-18T01:19:05.026 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:19:04 smithi090 ceph-mon[36989]: from='client.? 172.21.15.29:0/4124482494' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-18T01:19:05.035 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:05.034+0000 7f019c818700 1 -- 172.21.15.29:0/1107796317 --> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f019802b3e0 con 0x7f01800692a0 2024-03-18T01:19:05.037 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:05.035+0000 7f0186ffd700 1 -- 172.21.15.29:0/1107796317 <== mgr.14208 v2:172.21.15.29:6800/1641454714 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+358 (secure 0 0 0) 0x7f019802b3e0 con 0x7f01800692a0 2024-03-18T01:19:05.037 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-03-18T01:19:05.037 INFO:teuthology.orchestra.run.smithi029.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af", 2024-03-18T01:19:05.037 INFO:teuthology.orchestra.run.smithi029.stdout: "in_progress": true, 2024-03-18T01:19:05.037 INFO:teuthology.orchestra.run.smithi029.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-18T01:19:05.037 INFO:teuthology.orchestra.run.smithi029.stdout: "services_complete": [], 2024-03-18T01:19:05.037 INFO:teuthology.orchestra.run.smithi029.stdout: "progress": "", 2024-03-18T01:19:05.037 INFO:teuthology.orchestra.run.smithi029.stdout: "message": "Doing first pull of quay.ceph.io/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af image", 2024-03-18T01:19:05.037 INFO:teuthology.orchestra.run.smithi029.stdout: "is_paused": false 2024-03-18T01:19:05.037 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-03-18T01:19:05.040 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:05.038+0000 7f019c818700 1 -- 172.21.15.29:0/1107796317 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f01800692a0 msgr2=0x7f018006b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:19:05.040 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:05.038+0000 7f019c818700 1 --2- 172.21.15.29:0/1107796317 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f01800692a0 0x7f018006b750 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f018c008fd0 tx=0x7f018c00b040 comp rx=0 tx=0).stop 2024-03-18T01:19:05.040 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:05.038+0000 7f019c818700 1 -- 172.21.15.29:0/1107796317 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f01980fbce0 msgr2=0x7f019810d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:19:05.040 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:05.038+0000 7f019c818700 1 --2- 172.21.15.29:0/1107796317 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f01980fbce0 0x7f019810d9c0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f0188000c00 tx=0x7f0188009e30 comp rx=0 tx=0).stop 2024-03-18T01:19:05.040 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:05.039+0000 7f019c818700 1 -- 172.21.15.29:0/1107796317 shutdown_connections 2024-03-18T01:19:05.041 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:05.039+0000 7f019c818700 1 --2- 172.21.15.29:0/1107796317 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f01800692a0 0x7f018006b750 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:19:05.041 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:05.039+0000 7f019c818700 1 --2- 172.21.15.29:0/1107796317 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f01980fbce0 0x7f019810d9c0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:19:05.041 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:05.039+0000 7f019c818700 1 --2- 172.21.15.29:0/1107796317 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f01980fcf90 0x7f019810df00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:19:05.041 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:05.039+0000 7f019c818700 1 -- 172.21.15.29:0/1107796317 >> 172.21.15.29:0/1107796317 conn(0x7f01980f7440 msgr2=0x7f01980663c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:19:05.041 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:05.039+0000 7f019c818700 1 -- 172.21.15.29:0/1107796317 shutdown_connections 2024-03-18T01:19:05.041 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:05.039+0000 7f019c818700 1 -- 172.21.15.29:0/1107796317 wait complete. 2024-03-18T01:19:05.167 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:05.165+0000 7f70ab9ba700 1 -- 172.21.15.29:0/2151492609 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f70a40fffc0 msgr2=0x7f70a41023a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:19:05.167 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:05.165+0000 7f70ab9ba700 1 --2- 172.21.15.29:0/2151492609 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f70a40fffc0 0x7f70a41023a0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f7094004940 tx=0x7f7094009e30 comp rx=0 tx=0).stop 2024-03-18T01:19:05.167 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:05.165+0000 7f70ab9ba700 1 -- 172.21.15.29:0/2151492609 shutdown_connections 2024-03-18T01:19:05.167 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:05.165+0000 7f70ab9ba700 1 --2- 172.21.15.29:0/2151492609 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f70a40fffc0 0x7f70a41023a0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:19:05.167 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:05.165+0000 7f70ab9ba700 1 --2- 172.21.15.29:0/2151492609 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f70a4066220 0x7f70a40ffa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:19:05.167 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:05.165+0000 7f70ab9ba700 1 -- 172.21.15.29:0/2151492609 >> 172.21.15.29:0/2151492609 conn(0x7f70a40f7430 msgr2=0x7f70a40f9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:19:05.168 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:05.166+0000 7f70ab9ba700 1 -- 172.21.15.29:0/2151492609 shutdown_connections 2024-03-18T01:19:05.168 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:05.166+0000 7f70ab9ba700 1 -- 172.21.15.29:0/2151492609 wait complete. 2024-03-18T01:19:05.168 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:05.166+0000 7f70ab9ba700 1 Processor -- start 2024-03-18T01:19:05.168 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:05.167+0000 7f70ab9ba700 1 -- start start 2024-03-18T01:19:05.168 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:05.167+0000 7f70ab9ba700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f70a4066220 0x7f70a410da50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:19:05.168 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:05.167+0000 7f70ab9ba700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f70a40fffc0 0x7f70a410df90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:19:05.168 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:05.167+0000 7f70ab9ba700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70a410e590 con 0x7f70a40fffc0 2024-03-18T01:19:05.169 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:05.167+0000 7f70ab9ba700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70a410ed90 con 0x7f70a4066220 2024-03-18T01:19:05.170 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:05.167+0000 7f70a8f55700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f70a40fffc0 0x7f70a410df90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:19:05.170 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:05.167+0000 7f70a9756700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f70a4066220 0x7f70a410da50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:19:05.170 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:05.167+0000 7f70a8f55700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f70a40fffc0 0x7f70a410df90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60544/0 (socket says 172.21.15.29:60544) 2024-03-18T01:19:05.170 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:05.167+0000 7f70a8f55700 1 -- 172.21.15.29:0/1100514509 learned_addr learned my addr 172.21.15.29:0/1100514509 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-18T01:19:05.170 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:05.167+0000 7f70a8f55700 1 -- 172.21.15.29:0/1100514509 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f70a4066220 msgr2=0x7f70a410da50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:19:05.171 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:05.167+0000 7f70a8f55700 1 --2- 172.21.15.29:0/1100514509 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f70a4066220 0x7f70a410da50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:19:05.171 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:05.167+0000 7f70a8f55700 1 -- 172.21.15.29:0/1100514509 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7094005040 con 0x7f70a40fffc0 2024-03-18T01:19:05.171 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:05.168+0000 7f70a8f55700 1 --2- 172.21.15.29:0/1100514509 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f70a40fffc0 0x7f70a410df90 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f7094013870 tx=0x7f7094009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:19:05.171 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:05.168+0000 7f7099ffb700 1 -- 172.21.15.29:0/1100514509 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f709400e040 con 0x7f70a40fffc0 2024-03-18T01:19:05.171 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:05.168+0000 7f70ab9ba700 1 -- 172.21.15.29:0/1100514509 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70a410f010 con 0x7f70a40fffc0 2024-03-18T01:19:05.171 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:05.168+0000 7f7099ffb700 1 -- 172.21.15.29:0/1100514509 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7094014e40 con 0x7f70a40fffc0 2024-03-18T01:19:05.172 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:05.168+0000 7f7099ffb700 1 -- 172.21.15.29:0/1100514509 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7094024840 con 0x7f70a40fffc0 2024-03-18T01:19:05.172 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:05.168+0000 7f70ab9ba700 1 -- 172.21.15.29:0/1100514509 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70a410f550 con 0x7f70a40fffc0 2024-03-18T01:19:05.172 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:05.170+0000 7f70ab9ba700 1 -- 172.21.15.29:0/1100514509 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70a405f5f0 con 0x7f70a40fffc0 2024-03-18T01:19:05.172 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:05.170+0000 7f7099ffb700 1 -- 172.21.15.29:0/1100514509 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f7094022020 con 0x7f70a40fffc0 2024-03-18T01:19:05.172 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:05.171+0000 7f7099ffb700 1 --2- 172.21.15.29:0/1100514509 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f70900692b0 0x7f709006b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:19:05.173 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:05.171+0000 7f7099ffb700 1 -- 172.21.15.29:0/1100514509 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f709408c1c0 con 0x7f70a40fffc0 2024-03-18T01:19:05.173 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:05.172+0000 7f70a9756700 1 --2- 172.21.15.29:0/1100514509 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f70900692b0 0x7f709006b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:19:05.174 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:05.173+0000 7f70a9756700 1 --2- 172.21.15.29:0/1100514509 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f70900692b0 0x7f709006b760 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f70a0009b80 tx=0x7f70a000b040 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:19:05.177 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:05.176+0000 7f7099ffb700 1 -- 172.21.15.29:0/1100514509 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175520 (secure 0 0 0) 0x7f709405bf30 con 0x7f70a40fffc0 2024-03-18T01:19:05.604 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:05.602+0000 7f70ab9ba700 1 -- 172.21.15.29:0/1100514509 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f70a404bed0 con 0x7f70a40fffc0 2024-03-18T01:19:05.604 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:05.602+0000 7f7099ffb700 1 -- 172.21.15.29:0/1100514509 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+201 (secure 0 0 0) 0x7f709405bac0 con 0x7f70a40fffc0 2024-03-18T01:19:05.604 INFO:teuthology.orchestra.run.smithi029.stdout:HEALTH_WARN 1 filesystem with deprecated feature inline_data 2024-03-18T01:19:05.604 INFO:teuthology.orchestra.run.smithi029.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-03-18T01:19:05.604 INFO:teuthology.orchestra.run.smithi029.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-03-18T01:19:05.606 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:05.605+0000 7f70ab9ba700 1 -- 172.21.15.29:0/1100514509 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f70900692b0 msgr2=0x7f709006b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:19:05.607 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:05.605+0000 7f70ab9ba700 1 --2- 172.21.15.29:0/1100514509 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f70900692b0 0x7f709006b760 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f70a0009b80 tx=0x7f70a000b040 comp rx=0 tx=0).stop 2024-03-18T01:19:05.607 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:05.605+0000 7f70ab9ba700 1 -- 172.21.15.29:0/1100514509 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f70a40fffc0 msgr2=0x7f70a410df90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:19:05.607 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:05.605+0000 7f70ab9ba700 1 --2- 172.21.15.29:0/1100514509 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f70a40fffc0 0x7f70a410df90 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f7094013870 tx=0x7f7094009e30 comp rx=0 tx=0).stop 2024-03-18T01:19:05.607 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:05.605+0000 7f70ab9ba700 1 -- 172.21.15.29:0/1100514509 shutdown_connections 2024-03-18T01:19:05.607 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:05.605+0000 7f70ab9ba700 1 --2- 172.21.15.29:0/1100514509 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f70900692b0 0x7f709006b760 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:19:05.607 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:05.605+0000 7f70ab9ba700 1 --2- 172.21.15.29:0/1100514509 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f70a4066220 0x7f70a410da50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:19:05.607 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:05.605+0000 7f70ab9ba700 1 --2- 172.21.15.29:0/1100514509 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f70a40fffc0 0x7f70a410df90 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:19:05.607 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:05.605+0000 7f70ab9ba700 1 -- 172.21.15.29:0/1100514509 >> 172.21.15.29:0/1100514509 conn(0x7f70a40f7430 msgr2=0x7f70a40f7850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:19:05.607 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:05.605+0000 7f70ab9ba700 1 -- 172.21.15.29:0/1100514509 shutdown_connections 2024-03-18T01:19:05.607 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:05.605+0000 7f70ab9ba700 1 -- 172.21.15.29:0/1100514509 wait complete. 2024-03-18T01:19:06.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:19:06 smithi090 ceph-mon[36989]: pgmap v211: 65 pgs: 65 active+clean; 586 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-18T01:19:06.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:19:06 smithi090 ceph-mon[36989]: from='client.14580 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T01:19:06.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:19:06 smithi090 ceph-mon[36989]: from='client.? 172.21.15.29:0/1100514509' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-18T01:19:06.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:19:06 smithi029 ceph-mon[29331]: pgmap v211: 65 pgs: 65 active+clean; 586 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-18T01:19:06.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:19:06 smithi029 ceph-mon[29331]: from='client.14580 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T01:19:06.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:19:06 smithi029 ceph-mon[29331]: from='client.? 172.21.15.29:0/1100514509' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-18T01:19:08.426 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:19:08 smithi090 ceph-mon[36989]: pgmap v212: 65 pgs: 65 active+clean; 586 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-18T01:19:08.568 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:19:08 smithi029 ceph-mon[29331]: pgmap v212: 65 pgs: 65 active+clean; 586 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-18T01:19:10.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:19:10 smithi090 ceph-mon[36989]: pgmap v213: 65 pgs: 65 active+clean; 586 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-18T01:19:10.570 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:19:10 smithi029 ceph-mon[29331]: pgmap v213: 65 pgs: 65 active+clean; 586 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-18T01:19:11.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:19:11 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-18T01:19:11.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:19:11 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-18T01:19:12.525 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:19:12 smithi090 ceph-mon[36989]: pgmap v214: 65 pgs: 65 active+clean; 586 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-18T01:19:12.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:19:12 smithi029 ceph-mon[29331]: pgmap v214: 65 pgs: 65 active+clean; 586 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-18T01:19:14.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:19:14 smithi090 ceph-mon[36989]: pgmap v215: 65 pgs: 65 active+clean; 586 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-18T01:19:14.568 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:19:14 smithi029 ceph-mon[29331]: pgmap v215: 65 pgs: 65 active+clean; 586 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-18T01:19:16.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:19:16 smithi090 ceph-mon[36989]: pgmap v216: 65 pgs: 65 active+clean; 586 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-18T01:19:16.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:19:16 smithi029 ceph-mon[29331]: pgmap v216: 65 pgs: 65 active+clean; 586 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-18T01:19:18.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:19:18 smithi090 ceph-mon[36989]: pgmap v217: 65 pgs: 65 active+clean; 586 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-18T01:19:18.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:19:18 smithi029 ceph-mon[29331]: pgmap v217: 65 pgs: 65 active+clean; 586 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-18T01:19:20.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:19:20 smithi090 ceph-mon[36989]: pgmap v218: 65 pgs: 65 active+clean; 586 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-18T01:19:20.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:19:20 smithi029 ceph-mon[29331]: pgmap v218: 65 pgs: 65 active+clean; 586 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-18T01:19:22.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:19:22 smithi090 ceph-mon[36989]: pgmap v219: 65 pgs: 65 active+clean; 589 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.4 KiB/s rd, 426 B/s wr, 4 op/s 2024-03-18T01:19:22.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:19:22 smithi029 ceph-mon[29331]: pgmap v219: 65 pgs: 65 active+clean; 589 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.4 KiB/s rd, 426 B/s wr, 4 op/s 2024-03-18T01:19:24.061 INFO:tasks.workunit.client.1.smithi090.stderr:Note: switching to 'c382027d5c66ef4e0aa1131ea3e8080c658d3184'. 2024-03-18T01:19:24.061 INFO:tasks.workunit.client.1.smithi090.stderr: 2024-03-18T01:19:24.061 INFO:tasks.workunit.client.1.smithi090.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-03-18T01:19:24.062 INFO:tasks.workunit.client.1.smithi090.stderr:changes and commit them, and you can discard any commits you make in this 2024-03-18T01:19:24.062 INFO:tasks.workunit.client.1.smithi090.stderr:state without impacting any branches by switching back to a branch. 2024-03-18T01:19:24.062 INFO:tasks.workunit.client.1.smithi090.stderr: 2024-03-18T01:19:24.062 INFO:tasks.workunit.client.1.smithi090.stderr:If you want to create a new branch to retain commits you create, you may 2024-03-18T01:19:24.062 INFO:tasks.workunit.client.1.smithi090.stderr:do so (now or later) by using -c with the switch command. Example: 2024-03-18T01:19:24.062 INFO:tasks.workunit.client.1.smithi090.stderr: 2024-03-18T01:19:24.062 INFO:tasks.workunit.client.1.smithi090.stderr: git switch -c 2024-03-18T01:19:24.062 INFO:tasks.workunit.client.1.smithi090.stderr: 2024-03-18T01:19:24.062 INFO:tasks.workunit.client.1.smithi090.stderr:Or undo this operation with: 2024-03-18T01:19:24.062 INFO:tasks.workunit.client.1.smithi090.stderr: 2024-03-18T01:19:24.062 INFO:tasks.workunit.client.1.smithi090.stderr: git switch - 2024-03-18T01:19:24.062 INFO:tasks.workunit.client.1.smithi090.stderr: 2024-03-18T01:19:24.062 INFO:tasks.workunit.client.1.smithi090.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-03-18T01:19:24.063 INFO:tasks.workunit.client.1.smithi090.stderr: 2024-03-18T01:19:24.063 INFO:tasks.workunit.client.1.smithi090.stderr:HEAD is now at c382027d5c6 ignorelist mon down 2024-03-18T01:19:24.066 DEBUG:teuthology.orchestra.run.smithi090:> cd -- /home/ubuntu/cephtest/clone.client.1/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.1 2024-03-18T01:19:24.134 INFO:tasks.workunit.client.1.smithi090.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-03-18T01:19:24.135 INFO:tasks.workunit.client.1.smithi090.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/direct_io' 2024-03-18T01:19:24.136 INFO:tasks.workunit.client.1.smithi090.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-03-18T01:19:24.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:19:24 smithi090 ceph-mon[36989]: pgmap v220: 65 pgs: 65 active+clean; 589 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.4 KiB/s rd, 511 B/s wr, 4 op/s 2024-03-18T01:19:24.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:19:24 smithi029 ceph-mon[29331]: pgmap v220: 65 pgs: 65 active+clean; 589 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.4 KiB/s rd, 511 B/s wr, 4 op/s 2024-03-18T01:19:25.060 INFO:tasks.workunit.client.1.smithi090.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-03-18T01:19:25.159 INFO:tasks.workunit.client.1.smithi090.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-03-18T01:19:25.200 INFO:tasks.workunit.client.1.smithi090.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/direct_io' 2024-03-18T01:19:25.202 INFO:tasks.workunit.client.1.smithi090.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/fs' 2024-03-18T01:19:25.202 INFO:tasks.workunit.client.1.smithi090.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-03-18T01:19:25.241 INFO:tasks.workunit.client.1.smithi090.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/fs' 2024-03-18T01:19:25.244 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-03-18T01:19:25.244 DEBUG:teuthology.orchestra.run.smithi090:> dd if=/home/ubuntu/cephtest/workunits.list.client.1 of=/dev/stdout 2024-03-18T01:19:25.299 INFO:tasks.workunit:Running workunits matching suites/fsstress.sh on client.1... 2024-03-18T01:19:25.302 INFO:tasks.workunit:Running workunit suites/fsstress.sh... 2024-03-18T01:19:25.302 DEBUG:teuthology.orchestra.run.smithi090:workunit test suites/fsstress.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=c382027d5c66ef4e0aa1131ea3e8080c658d3184 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.1/qa/workunits/suites/fsstress.sh 2024-03-18T01:19:25.362 INFO:tasks.workunit.client.1.smithi090.stderr:+ mkdir -p fsstress 2024-03-18T01:19:25.364 INFO:tasks.workunit.client.1.smithi090.stderr:+ pushd fsstress 2024-03-18T01:19:25.364 INFO:tasks.workunit.client.1.smithi090.stdout:~/cephtest/mnt.1/client.1/tmp/fsstress ~/cephtest/mnt.1/client.1/tmp 2024-03-18T01:19:25.364 INFO:tasks.workunit.client.1.smithi090.stderr:+ wget -q -O ltp-full.tgz http://download.ceph.com/qa/ltp-full-20091231.tgz 2024-03-18T01:19:26.191 INFO:tasks.workunit.client.1.smithi090.stderr:+ tar xzf ltp-full.tgz 2024-03-18T01:19:26.450 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:19:26 smithi029 ceph-mon[29331]: pgmap v221: 65 pgs: 65 active+clean; 589 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.4 KiB/s rd, 511 B/s wr, 4 op/s 2024-03-18T01:19:26.450 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:19:26 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-18T01:19:26.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:19:26 smithi090 ceph-mon[36989]: pgmap v221: 65 pgs: 65 active+clean; 589 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.4 KiB/s rd, 511 B/s wr, 4 op/s 2024-03-18T01:19:26.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:19:26 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-18T01:19:28.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:19:28 smithi090 ceph-mon[36989]: pgmap v222: 65 pgs: 65 active+clean; 589 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.0 KiB/s rd, 597 B/s wr, 4 op/s 2024-03-18T01:19:28.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:19:28 smithi029 ceph-mon[29331]: pgmap v222: 65 pgs: 65 active+clean; 589 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.0 KiB/s rd, 597 B/s wr, 4 op/s 2024-03-18T01:19:29.552 INFO:tasks.workunit.client.0.smithi029.stderr:Note: switching to 'c382027d5c66ef4e0aa1131ea3e8080c658d3184'. 2024-03-18T01:19:29.552 INFO:tasks.workunit.client.0.smithi029.stderr: 2024-03-18T01:19:29.552 INFO:tasks.workunit.client.0.smithi029.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-03-18T01:19:29.552 INFO:tasks.workunit.client.0.smithi029.stderr:changes and commit them, and you can discard any commits you make in this 2024-03-18T01:19:29.552 INFO:tasks.workunit.client.0.smithi029.stderr:state without impacting any branches by switching back to a branch. 2024-03-18T01:19:29.552 INFO:tasks.workunit.client.0.smithi029.stderr: 2024-03-18T01:19:29.552 INFO:tasks.workunit.client.0.smithi029.stderr:If you want to create a new branch to retain commits you create, you may 2024-03-18T01:19:29.552 INFO:tasks.workunit.client.0.smithi029.stderr:do so (now or later) by using -c with the switch command. Example: 2024-03-18T01:19:29.552 INFO:tasks.workunit.client.0.smithi029.stderr: 2024-03-18T01:19:29.552 INFO:tasks.workunit.client.0.smithi029.stderr: git switch -c 2024-03-18T01:19:29.552 INFO:tasks.workunit.client.0.smithi029.stderr: 2024-03-18T01:19:29.552 INFO:tasks.workunit.client.0.smithi029.stderr:Or undo this operation with: 2024-03-18T01:19:29.553 INFO:tasks.workunit.client.0.smithi029.stderr: 2024-03-18T01:19:29.553 INFO:tasks.workunit.client.0.smithi029.stderr: git switch - 2024-03-18T01:19:29.553 INFO:tasks.workunit.client.0.smithi029.stderr: 2024-03-18T01:19:29.553 INFO:tasks.workunit.client.0.smithi029.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-03-18T01:19:29.553 INFO:tasks.workunit.client.0.smithi029.stderr: 2024-03-18T01:19:29.553 INFO:tasks.workunit.client.0.smithi029.stderr:HEAD is now at c382027d5c6 ignorelist mon down 2024-03-18T01:19:29.560 DEBUG:teuthology.orchestra.run.smithi029:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-03-18T01:19:29.589 INFO:tasks.workunit.client.0.smithi029.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-03-18T01:19:29.592 INFO:tasks.workunit.client.0.smithi029.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-03-18T01:19:29.592 INFO:tasks.workunit.client.0.smithi029.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-03-18T01:19:30.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:19:30 smithi090 ceph-mon[36989]: pgmap v223: 65 pgs: 65 active+clean; 589 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.4 KiB/s rd, 597 B/s wr, 4 op/s 2024-03-18T01:19:30.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:19:30 smithi029 ceph-mon[29331]: pgmap v223: 65 pgs: 65 active+clean; 589 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.4 KiB/s rd, 597 B/s wr, 4 op/s 2024-03-18T01:19:32.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:19:32 smithi090 ceph-mon[36989]: pgmap v224: 65 pgs: 65 active+clean; 4.6 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 343 KiB/s rd, 342 KiB/s wr, 5 op/s 2024-03-18T01:19:32.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:19:32 smithi029 ceph-mon[29331]: pgmap v224: 65 pgs: 65 active+clean; 4.6 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 343 KiB/s rd, 342 KiB/s wr, 5 op/s 2024-03-18T01:19:34.441 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:19:34 smithi029 ceph-mon[29331]: pgmap v225: 65 pgs: 65 active+clean; 28 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 684 KiB/s rd, 2.3 MiB/s wr, 147 op/s 2024-03-18T01:19:34.441 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:19:34 smithi029 ceph-mon[29331]: fsmap cephfs:2 {0=cephfs.smithi029.brxqeh=up:active,1=cephfs.smithi090.bohder=up:active} 2 up:standby-replay 2024-03-18T01:19:34.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:19:34 smithi090 ceph-mon[36989]: pgmap v225: 65 pgs: 65 active+clean; 28 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 684 KiB/s rd, 2.3 MiB/s wr, 147 op/s 2024-03-18T01:19:34.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:19:34 smithi090 ceph-mon[36989]: fsmap cephfs:2 {0=cephfs.smithi029.brxqeh=up:active,1=cephfs.smithi090.bohder=up:active} 2 up:standby-replay 2024-03-18T01:19:34.549 INFO:tasks.workunit.client.0.smithi029.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-03-18T01:19:34.692 INFO:tasks.workunit.client.0.smithi029.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-03-18T01:19:34.740 INFO:tasks.workunit.client.0.smithi029.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-03-18T01:19:34.741 INFO:tasks.workunit.client.0.smithi029.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-03-18T01:19:34.741 INFO:tasks.workunit.client.0.smithi029.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-03-18T01:19:34.783 INFO:tasks.workunit.client.0.smithi029.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-03-18T01:19:34.786 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-03-18T01:19:34.786 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-03-18T01:19:34.842 INFO:tasks.workunit:Running workunits matching suites/fsstress.sh on client.0... 2024-03-18T01:19:34.844 INFO:tasks.workunit:Running workunit suites/fsstress.sh... 2024-03-18T01:19:34.844 DEBUG:teuthology.orchestra.run.smithi029:workunit test suites/fsstress.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=c382027d5c66ef4e0aa1131ea3e8080c658d3184 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsstress.sh 2024-03-18T01:19:34.906 INFO:tasks.workunit.client.0.smithi029.stderr:+ mkdir -p fsstress 2024-03-18T01:19:34.908 INFO:tasks.workunit.client.0.smithi029.stderr:+ pushd fsstress 2024-03-18T01:19:34.908 INFO:tasks.workunit.client.0.smithi029.stdout:~/cephtest/mnt.0/client.0/tmp/fsstress ~/cephtest/mnt.0/client.0/tmp 2024-03-18T01:19:34.909 INFO:tasks.workunit.client.0.smithi029.stderr:+ wget -q -O ltp-full.tgz http://download.ceph.com/qa/ltp-full-20091231.tgz 2024-03-18T01:19:35.480 INFO:tasks.workunit.client.0.smithi029.stderr:+ tar xzf ltp-full.tgz 2024-03-18T01:19:35.734 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:35.733+0000 7f65ba7ea700 1 -- 172.21.15.29:0/3236546730 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f65b40fbcd0 msgr2=0x7f65b40fc0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:19:35.735 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:35.733+0000 7f65ba7ea700 1 --2- 172.21.15.29:0/3236546730 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f65b40fbcd0 0x7f65b40fc0e0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f65a4004940 tx=0x7f65a4009e30 comp rx=0 tx=0).stop 2024-03-18T01:19:35.735 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:35.734+0000 7f65ba7ea700 1 -- 172.21.15.29:0/3236546730 shutdown_connections 2024-03-18T01:19:35.735 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:35.734+0000 7f65ba7ea700 1 --2- 172.21.15.29:0/3236546730 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f65b40fcf80 0x7f65b40fd3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:19:35.735 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:35.734+0000 7f65ba7ea700 1 --2- 172.21.15.29:0/3236546730 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f65b40fbcd0 0x7f65b40fc0e0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:19:35.735 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:35.734+0000 7f65ba7ea700 1 -- 172.21.15.29:0/3236546730 >> 172.21.15.29:0/3236546730 conn(0x7f65b40f7430 msgr2=0x7f65b40f9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:19:35.735 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:35.734+0000 7f65ba7ea700 1 -- 172.21.15.29:0/3236546730 shutdown_connections 2024-03-18T01:19:35.735 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:35.734+0000 7f65ba7ea700 1 -- 172.21.15.29:0/3236546730 wait complete. 2024-03-18T01:19:35.735 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:35.734+0000 7f65ba7ea700 1 Processor -- start 2024-03-18T01:19:35.735 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:35.734+0000 7f65ba7ea700 1 -- start start 2024-03-18T01:19:35.736 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:35.735+0000 7f65ba7ea700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f65b40fbcd0 0x7f65b410daa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:19:35.736 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:35.735+0000 7f65ba7ea700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f65b40fcf80 0x7f65b410dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:19:35.736 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:35.735+0000 7f65ba7ea700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65b410e5e0 con 0x7f65b40fbcd0 2024-03-18T01:19:35.737 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:35.735+0000 7f65ba7ea700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65b410ed90 con 0x7f65b40fcf80 2024-03-18T01:19:35.737 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:35.735+0000 7f65b3fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f65b40fbcd0 0x7f65b410daa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:19:35.737 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:35.735+0000 7f65b3fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f65b40fbcd0 0x7f65b410daa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:55912/0 (socket says 172.21.15.29:55912) 2024-03-18T01:19:35.737 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:35.735+0000 7f65b3fff700 1 -- 172.21.15.29:0/3519185898 learned_addr learned my addr 172.21.15.29:0/3519185898 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-18T01:19:35.737 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:35.735+0000 7f65b37fe700 1 --2- 172.21.15.29:0/3519185898 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f65b40fcf80 0x7f65b410dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:19:35.737 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:35.735+0000 7f65b37fe700 1 -- 172.21.15.29:0/3519185898 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f65b40fbcd0 msgr2=0x7f65b410daa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:19:35.737 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:35.735+0000 7f65b37fe700 1 --2- 172.21.15.29:0/3519185898 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f65b40fbcd0 0x7f65b410daa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:19:35.737 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:35.735+0000 7f65b37fe700 1 -- 172.21.15.29:0/3519185898 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65a4005040 con 0x7f65b40fcf80 2024-03-18T01:19:35.737 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:35.735+0000 7f65b3fff700 1 --2- 172.21.15.29:0/3519185898 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f65b40fbcd0 0x7f65b410daa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T01:19:35.737 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:35.735+0000 7f65b37fe700 1 --2- 172.21.15.29:0/3519185898 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f65b40fcf80 0x7f65b410dfe0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f65a8002820 tx=0x7f65a8004570 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:19:35.737 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:35.736+0000 7f65b0ff9700 1 -- 172.21.15.29:0/3519185898 <== mon.1 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f65a8007cc0 con 0x7f65b40fcf80 2024-03-18T01:19:35.737 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:35.736+0000 7f65ba7ea700 1 -- 172.21.15.29:0/3519185898 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65b410f070 con 0x7f65b40fcf80 2024-03-18T01:19:35.737 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:35.736+0000 7f65b0ff9700 1 -- 172.21.15.29:0/3519185898 <== mon.1 v2:172.21.15.90:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f65a800f040 con 0x7f65b40fcf80 2024-03-18T01:19:35.737 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:35.736+0000 7f65b0ff9700 1 -- 172.21.15.29:0/3519185898 <== mon.1 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f65a8013920 con 0x7f65b40fcf80 2024-03-18T01:19:35.738 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:35.736+0000 7f65ba7ea700 1 -- 172.21.15.29:0/3519185898 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65b410f5b0 con 0x7f65b40fcf80 2024-03-18T01:19:35.738 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:35.737+0000 7f65ba7ea700 1 -- 172.21.15.29:0/3519185898 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65b4101b10 con 0x7f65b40fcf80 2024-03-18T01:19:35.738 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:35.737+0000 7f65b0ff9700 1 -- 172.21.15.29:0/3519185898 <== mon.1 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f65a8013b80 con 0x7f65b40fcf80 2024-03-18T01:19:35.739 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:35.737+0000 7f65b0ff9700 1 --2- 172.21.15.29:0/3519185898 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f659c0692b0 0x7f659c06b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:19:35.739 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:35.737+0000 7f65b0ff9700 1 -- 172.21.15.29:0/3519185898 <== mon.1 v2:172.21.15.90:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f65a8086ec0 con 0x7f65b40fcf80 2024-03-18T01:19:35.739 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:35.737+0000 7f65b3fff700 1 --2- 172.21.15.29:0/3519185898 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f659c0692b0 0x7f659c06b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:19:35.739 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:35.738+0000 7f65b3fff700 1 --2- 172.21.15.29:0/3519185898 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f659c0692b0 0x7f659c06b760 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f65a400b3b0 tx=0x7f65a4009e30 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:19:35.742 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:35.742+0000 7f65b0ff9700 1 -- 172.21.15.29:0/3519185898 <== mon.1 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175520 (secure 0 0 0) 0x7f65a8056c00 con 0x7f65b40fcf80 2024-03-18T01:19:35.982 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:35.982+0000 7f65ba7ea700 1 -- 172.21.15.29:0/3519185898 --> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f65b410f820 con 0x7f659c0692b0 2024-03-18T01:19:35.983 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:35.983+0000 7f65b0ff9700 1 -- 172.21.15.29:0/3519185898 <== mgr.14208 v2:172.21.15.29:6800/1641454714 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+349 (secure 0 0 0) 0x7f65b410f820 con 0x7f659c0692b0 2024-03-18T01:19:35.985 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:35.985+0000 7f65ba7ea700 1 -- 172.21.15.29:0/3519185898 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f659c0692b0 msgr2=0x7f659c06b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:19:35.985 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:35.985+0000 7f65ba7ea700 1 --2- 172.21.15.29:0/3519185898 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f659c0692b0 0x7f659c06b760 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f65a400b3b0 tx=0x7f65a4009e30 comp rx=0 tx=0).stop 2024-03-18T01:19:35.986 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:35.985+0000 7f65ba7ea700 1 -- 172.21.15.29:0/3519185898 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f65b40fcf80 msgr2=0x7f65b410dfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:19:35.986 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:35.985+0000 7f65ba7ea700 1 --2- 172.21.15.29:0/3519185898 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f65b40fcf80 0x7f65b410dfe0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f65a8002820 tx=0x7f65a8004570 comp rx=0 tx=0).stop 2024-03-18T01:19:35.986 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:35.985+0000 7f65ba7ea700 1 -- 172.21.15.29:0/3519185898 shutdown_connections 2024-03-18T01:19:35.986 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:35.985+0000 7f65ba7ea700 1 --2- 172.21.15.29:0/3519185898 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f659c0692b0 0x7f659c06b760 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:19:35.986 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:35.985+0000 7f65ba7ea700 1 --2- 172.21.15.29:0/3519185898 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f65b40fbcd0 0x7f65b410daa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:19:35.986 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:35.985+0000 7f65ba7ea700 1 --2- 172.21.15.29:0/3519185898 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f65b40fcf80 0x7f65b410dfe0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:19:35.986 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:35.985+0000 7f65ba7ea700 1 -- 172.21.15.29:0/3519185898 >> 172.21.15.29:0/3519185898 conn(0x7f65b40f7430 msgr2=0x7f65b4065b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:19:35.986 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:35.985+0000 7f65ba7ea700 1 -- 172.21.15.29:0/3519185898 shutdown_connections 2024-03-18T01:19:35.986 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:35.985+0000 7f65ba7ea700 1 -- 172.21.15.29:0/3519185898 wait complete. 2024-03-18T01:19:35.996 INFO:teuthology.orchestra.run.smithi029.stdout:true 2024-03-18T01:19:36.092 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:36.091+0000 7f5a33fc9700 1 -- 172.21.15.29:0/22661997 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5a2c0fbcd0 msgr2=0x7f5a2c0fc0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:19:36.093 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:36.091+0000 7f5a33fc9700 1 --2- 172.21.15.29:0/22661997 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5a2c0fbcd0 0x7f5a2c0fc0e0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f5a28004940 tx=0x7f5a28009e30 comp rx=0 tx=0).stop 2024-03-18T01:19:36.093 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:36.091+0000 7f5a33fc9700 1 -- 172.21.15.29:0/22661997 shutdown_connections 2024-03-18T01:19:36.093 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:36.091+0000 7f5a33fc9700 1 --2- 172.21.15.29:0/22661997 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5a2c0fcf80 0x7f5a2c0fd3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:19:36.093 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:36.091+0000 7f5a33fc9700 1 --2- 172.21.15.29:0/22661997 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5a2c0fbcd0 0x7f5a2c0fc0e0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:19:36.093 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:36.091+0000 7f5a33fc9700 1 -- 172.21.15.29:0/22661997 >> 172.21.15.29:0/22661997 conn(0x7f5a2c0f7430 msgr2=0x7f5a2c0f9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:19:36.093 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:36.092+0000 7f5a33fc9700 1 -- 172.21.15.29:0/22661997 shutdown_connections 2024-03-18T01:19:36.093 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:36.092+0000 7f5a33fc9700 1 -- 172.21.15.29:0/22661997 wait complete. 2024-03-18T01:19:36.093 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:36.092+0000 7f5a33fc9700 1 Processor -- start 2024-03-18T01:19:36.093 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:36.092+0000 7f5a33fc9700 1 -- start start 2024-03-18T01:19:36.093 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:36.092+0000 7f5a33fc9700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5a2c0fbcd0 0x7f5a2c10d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:19:36.093 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:36.092+0000 7f5a33fc9700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5a2c0fcf80 0x7f5a2c10df00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:19:36.093 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:36.092+0000 7f5a33fc9700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a2c10e500 con 0x7f5a2c0fcf80 2024-03-18T01:19:36.094 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:36.092+0000 7f5a33fc9700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a2c10ed00 con 0x7f5a2c0fbcd0 2024-03-18T01:19:36.095 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:36.093+0000 7f5a31564700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5a2c0fcf80 0x7f5a2c10df00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:19:36.095 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:36.093+0000 7f5a31d65700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5a2c0fbcd0 0x7f5a2c10d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:19:36.095 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:36.093+0000 7f5a31564700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5a2c0fcf80 0x7f5a2c10df00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:55920/0 (socket says 172.21.15.29:55920) 2024-03-18T01:19:36.095 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:36.093+0000 7f5a31564700 1 -- 172.21.15.29:0/2123248566 learned_addr learned my addr 172.21.15.29:0/2123248566 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-18T01:19:36.095 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:36.093+0000 7f5a31d65700 1 -- 172.21.15.29:0/2123248566 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5a2c0fcf80 msgr2=0x7f5a2c10df00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:19:36.095 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:36.093+0000 7f5a31d65700 1 --2- 172.21.15.29:0/2123248566 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5a2c0fcf80 0x7f5a2c10df00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:19:36.095 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:36.093+0000 7f5a31d65700 1 -- 172.21.15.29:0/2123248566 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a28005040 con 0x7f5a2c0fbcd0 2024-03-18T01:19:36.095 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:36.093+0000 7f5a31d65700 1 --2- 172.21.15.29:0/2123248566 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5a2c0fbcd0 0x7f5a2c10d9c0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f5a28013870 tx=0x7f5a28009e30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:19:36.095 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:36.093+0000 7f5a227fc700 1 -- 172.21.15.29:0/2123248566 <== mon.1 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5a2800e030 con 0x7f5a2c0fbcd0 2024-03-18T01:19:36.095 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:36.093+0000 7f5a33fc9700 1 -- 172.21.15.29:0/2123248566 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a2c10ef80 con 0x7f5a2c0fbcd0 2024-03-18T01:19:36.095 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:36.093+0000 7f5a227fc700 1 -- 172.21.15.29:0/2123248566 <== mon.1 v2:172.21.15.90:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5a28009430 con 0x7f5a2c0fbcd0 2024-03-18T01:19:36.096 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:36.093+0000 7f5a227fc700 1 -- 172.21.15.29:0/2123248566 <== mon.1 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5a2800b410 con 0x7f5a2c0fbcd0 2024-03-18T01:19:36.096 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:36.094+0000 7f5a33fc9700 1 -- 172.21.15.29:0/2123248566 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a2c10f4c0 con 0x7f5a2c0fbcd0 2024-03-18T01:19:36.096 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:36.095+0000 7f5a33fc9700 1 -- 172.21.15.29:0/2123248566 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a2c05f5f0 con 0x7f5a2c0fbcd0 2024-03-18T01:19:36.096 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:36.095+0000 7f5a227fc700 1 -- 172.21.15.29:0/2123248566 <== mon.1 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f5a2801c750 con 0x7f5a2c0fbcd0 2024-03-18T01:19:36.097 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:36.096+0000 7f5a227fc700 1 --2- 172.21.15.29:0/2123248566 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f5a1806d6b0 0x7f5a1806fb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:19:36.097 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:36.096+0000 7f5a227fc700 1 -- 172.21.15.29:0/2123248566 <== mon.1 v2:172.21.15.90:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f5a28088400 con 0x7f5a2c0fbcd0 2024-03-18T01:19:36.097 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:36.096+0000 7f5a31564700 1 --2- 172.21.15.29:0/2123248566 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f5a1806d6b0 0x7f5a1806fb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:19:36.097 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:36.096+0000 7f5a31564700 1 --2- 172.21.15.29:0/2123248566 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f5a1806d6b0 0x7f5a1806fb60 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f5a1c008fd0 tx=0x7f5a1c00b040 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:19:36.100 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:36.100+0000 7f5a227fc700 1 -- 172.21.15.29:0/2123248566 <== mon.1 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175520 (secure 0 0 0) 0x7f5a28058090 con 0x7f5a2c0fbcd0 2024-03-18T01:19:36.345 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:36.344+0000 7f5a33fc9700 1 -- 172.21.15.29:0/2123248566 --> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f5a2c02b340 con 0x7f5a1806d6b0 2024-03-18T01:19:36.345 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:19:36 smithi029 ceph-mon[29331]: pgmap v226: 65 pgs: 65 active+clean; 45 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.1 MiB/s rd, 3.9 MiB/s wr, 315 op/s 2024-03-18T01:19:36.345 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:19:36 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:19:36.345 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:19:36 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T01:19:36.345 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:19:36 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:19:36.346 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:36.346+0000 7f5a227fc700 1 -- 172.21.15.29:0/2123248566 <== mgr.14208 v2:172.21.15.29:6800/1641454714 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+447 (secure 0 0 0) 0x7f5a2c02b340 con 0x7f5a1806d6b0 2024-03-18T01:19:36.348 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:36.348+0000 7f5a33fc9700 1 -- 172.21.15.29:0/2123248566 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f5a1806d6b0 msgr2=0x7f5a1806fb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:19:36.349 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:36.348+0000 7f5a33fc9700 1 --2- 172.21.15.29:0/2123248566 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f5a1806d6b0 0x7f5a1806fb60 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f5a1c008fd0 tx=0x7f5a1c00b040 comp rx=0 tx=0).stop 2024-03-18T01:19:36.349 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:36.348+0000 7f5a33fc9700 1 -- 172.21.15.29:0/2123248566 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5a2c0fbcd0 msgr2=0x7f5a2c10d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:19:36.349 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:36.348+0000 7f5a33fc9700 1 --2- 172.21.15.29:0/2123248566 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5a2c0fbcd0 0x7f5a2c10d9c0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f5a28013870 tx=0x7f5a28009e30 comp rx=0 tx=0).stop 2024-03-18T01:19:36.349 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:36.348+0000 7f5a33fc9700 1 -- 172.21.15.29:0/2123248566 shutdown_connections 2024-03-18T01:19:36.349 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:36.348+0000 7f5a33fc9700 1 --2- 172.21.15.29:0/2123248566 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f5a1806d6b0 0x7f5a1806fb60 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:19:36.349 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:36.348+0000 7f5a33fc9700 1 --2- 172.21.15.29:0/2123248566 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5a2c0fbcd0 0x7f5a2c10d9c0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:19:36.349 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:36.348+0000 7f5a33fc9700 1 --2- 172.21.15.29:0/2123248566 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5a2c0fcf80 0x7f5a2c10df00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:19:36.349 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:36.348+0000 7f5a33fc9700 1 -- 172.21.15.29:0/2123248566 >> 172.21.15.29:0/2123248566 conn(0x7f5a2c0f7430 msgr2=0x7f5a2c065b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:19:36.349 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:36.348+0000 7f5a33fc9700 1 -- 172.21.15.29:0/2123248566 shutdown_connections 2024-03-18T01:19:36.349 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:36.348+0000 7f5a33fc9700 1 -- 172.21.15.29:0/2123248566 wait complete. 2024-03-18T01:19:36.461 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:36.460+0000 7fed27758700 1 -- 172.21.15.29:0/3586153824 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fed200fcf80 msgr2=0x7fed200fd3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:19:36.461 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:36.460+0000 7fed27758700 1 --2- 172.21.15.29:0/3586153824 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fed200fcf80 0x7fed200fd3f0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fed10004940 tx=0x7fed10009e30 comp rx=0 tx=0).stop 2024-03-18T01:19:36.461 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:36.460+0000 7fed27758700 1 -- 172.21.15.29:0/3586153824 shutdown_connections 2024-03-18T01:19:36.461 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:36.460+0000 7fed27758700 1 --2- 172.21.15.29:0/3586153824 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fed200fcf80 0x7fed200fd3f0 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:19:36.461 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:36.460+0000 7fed27758700 1 --2- 172.21.15.29:0/3586153824 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fed200fbcd0 0x7fed200fc0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:19:36.461 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:36.460+0000 7fed27758700 1 -- 172.21.15.29:0/3586153824 >> 172.21.15.29:0/3586153824 conn(0x7fed200f7430 msgr2=0x7fed200f9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:19:36.462 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:36.461+0000 7fed27758700 1 -- 172.21.15.29:0/3586153824 shutdown_connections 2024-03-18T01:19:36.462 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:36.461+0000 7fed27758700 1 -- 172.21.15.29:0/3586153824 wait complete. 2024-03-18T01:19:36.463 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:36.461+0000 7fed27758700 1 Processor -- start 2024-03-18T01:19:36.463 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:36.461+0000 7fed27758700 1 -- start start 2024-03-18T01:19:36.463 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:36.462+0000 7fed27758700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fed200fbcd0 0x7fed2010d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:19:36.463 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:36.462+0000 7fed27758700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fed200fcf80 0x7fed2010df20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:19:36.464 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:36.462+0000 7fed27758700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed2010e520 con 0x7fed200fbcd0 2024-03-18T01:19:36.464 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:36.462+0000 7fed27758700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed2010ed20 con 0x7fed200fcf80 2024-03-18T01:19:36.464 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:36.462+0000 7fed254f4700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fed200fbcd0 0x7fed2010d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:19:36.464 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:36.462+0000 7fed254f4700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fed200fbcd0 0x7fed2010d9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:55952/0 (socket says 172.21.15.29:55952) 2024-03-18T01:19:36.464 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:36.462+0000 7fed254f4700 1 -- 172.21.15.29:0/1244741459 learned_addr learned my addr 172.21.15.29:0/1244741459 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-18T01:19:36.464 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:36.462+0000 7fed24cf3700 1 --2- 172.21.15.29:0/1244741459 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fed200fcf80 0x7fed2010df20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:19:36.464 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:36.462+0000 7fed254f4700 1 -- 172.21.15.29:0/1244741459 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fed200fcf80 msgr2=0x7fed2010df20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:19:36.464 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:36.462+0000 7fed254f4700 1 --2- 172.21.15.29:0/1244741459 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fed200fcf80 0x7fed2010df20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:19:36.464 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:36.462+0000 7fed254f4700 1 -- 172.21.15.29:0/1244741459 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed10005040 con 0x7fed200fbcd0 2024-03-18T01:19:36.464 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:36.463+0000 7fed24cf3700 1 --2- 172.21.15.29:0/1244741459 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fed200fcf80 0x7fed2010df20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T01:19:36.464 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:36.463+0000 7fed254f4700 1 --2- 172.21.15.29:0/1244741459 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fed200fbcd0 0x7fed2010d9e0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fed1c00eb40 tx=0x7fed1c00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:19:36.464 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:36.463+0000 7fed15ffb700 1 -- 172.21.15.29:0/1244741459 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fed1c007cc0 con 0x7fed200fbcd0 2024-03-18T01:19:36.464 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:36.463+0000 7fed27758700 1 -- 172.21.15.29:0/1244741459 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed2010efa0 con 0x7fed200fbcd0 2024-03-18T01:19:36.465 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:36.463+0000 7fed27758700 1 -- 172.21.15.29:0/1244741459 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed2010f4e0 con 0x7fed200fbcd0 2024-03-18T01:19:36.465 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:36.464+0000 7fed15ffb700 1 -- 172.21.15.29:0/1244741459 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fed1c00f040 con 0x7fed200fbcd0 2024-03-18T01:19:36.465 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:36.464+0000 7fed15ffb700 1 -- 172.21.15.29:0/1244741459 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fed1c00c990 con 0x7fed200fbcd0 2024-03-18T01:19:36.466 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:36.465+0000 7fed27758700 1 -- 172.21.15.29:0/1244741459 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed20101b10 con 0x7fed200fbcd0 2024-03-18T01:19:36.466 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:36.465+0000 7fed15ffb700 1 -- 172.21.15.29:0/1244741459 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7fed1c00caf0 con 0x7fed200fbcd0 2024-03-18T01:19:36.466 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:36.466+0000 7fed15ffb700 1 --2- 172.21.15.29:0/1244741459 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7fed0c0692a0 0x7fed0c06b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:19:36.466 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:36.466+0000 7fed15ffb700 1 -- 172.21.15.29:0/1244741459 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7fed1c086df0 con 0x7fed200fbcd0 2024-03-18T01:19:36.466 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:36.466+0000 7fed24cf3700 1 --2- 172.21.15.29:0/1244741459 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7fed0c0692a0 0x7fed0c06b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:19:36.467 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:36.467+0000 7fed24cf3700 1 --2- 172.21.15.29:0/1244741459 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7fed0c0692a0 0x7fed0c06b750 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fed10009b30 tx=0x7fed10009e30 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:19:36.471 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:36.471+0000 7fed15ffb700 1 -- 172.21.15.29:0/1244741459 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175520 (secure 0 0 0) 0x7fed1c04ddb0 con 0x7fed200fbcd0 2024-03-18T01:19:36.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:19:36 smithi090 ceph-mon[36989]: pgmap v226: 65 pgs: 65 active+clean; 45 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.1 MiB/s rd, 3.9 MiB/s wr, 315 op/s 2024-03-18T01:19:36.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:19:36 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:19:36.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:19:36 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T01:19:36.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:19:36 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' 2024-03-18T01:19:36.706 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:36.706+0000 7fed27758700 1 -- 172.21.15.29:0/1244741459 --> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fed20104930 con 0x7fed0c0692a0 2024-03-18T01:19:36.719 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:36.718+0000 7fed15ffb700 1 -- 172.21.15.29:0/1244741459 <== mgr.14208 v2:172.21.15.29:6800/1641454714 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3366 (secure 0 0 0) 0x7fed20104930 con 0x7fed0c0692a0 2024-03-18T01:19:36.719 INFO:teuthology.orchestra.run.smithi029.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-18T01:19:36.719 INFO:teuthology.orchestra.run.smithi029.stdout:alertmanager.smithi029 smithi029 *:9093,9094 running (5m) 64s ago 7m 24.9M - 0.25.0 c8568f914cd2 74aa94ac819f 2024-03-18T01:19:36.720 INFO:teuthology.orchestra.run.smithi029.stdout:crash.smithi029 smithi029 running (7m) 64s ago 7m 6660k - 17.2.7-684-g6716c9f3 19857db3290c a7a7a1f5abb1 2024-03-18T01:19:36.720 INFO:teuthology.orchestra.run.smithi029.stdout:crash.smithi090 smithi090 running (6m) 71s ago 6m 6665k - 17.2.7-684-g6716c9f3 19857db3290c a0efb58cf729 2024-03-18T01:19:36.720 INFO:teuthology.orchestra.run.smithi029.stdout:grafana.smithi029 smithi029 *:3000 running (5m) 64s ago 6m 86.6M - 9.4.7 954c08fa6188 875561d97b0e 2024-03-18T01:19:36.720 INFO:teuthology.orchestra.run.smithi029.stdout:mds.cephfs.smithi029.brxqeh smithi029 running (84s) 64s ago 83s 14.7M - 17.2.7-684-g6716c9f3 19857db3290c d569540253f3 2024-03-18T01:19:36.720 INFO:teuthology.orchestra.run.smithi029.stdout:mds.cephfs.smithi029.xuxvlr smithi029 running (90s) 64s ago 89s 11.3M - 17.2.7-684-g6716c9f3 19857db3290c 6d9efbc518c2 2024-03-18T01:19:36.720 INFO:teuthology.orchestra.run.smithi029.stdout:mds.cephfs.smithi090.bohder smithi090 running (87s) 71s ago 86s 14.4M - 17.2.7-684-g6716c9f3 19857db3290c 7726489bb8a0 2024-03-18T01:19:36.720 INFO:teuthology.orchestra.run.smithi029.stdout:mds.cephfs.smithi090.trsnxz smithi090 running (81s) 71s ago 81s 14.4M - 17.2.7-684-g6716c9f3 19857db3290c b4d306a4cb22 2024-03-18T01:19:36.720 INFO:teuthology.orchestra.run.smithi029.stdout:mgr.smithi029.qviwhl smithi029 *:9283 running (8m) 64s ago 8m 480M - 17.2.7-684-g6716c9f3 19857db3290c 3ff37450535a 2024-03-18T01:19:36.720 INFO:teuthology.orchestra.run.smithi029.stdout:mgr.smithi090.pnqedy smithi090 *:8443,9283 running (6m) 71s ago 6m 432M - 17.2.7-684-g6716c9f3 19857db3290c 68666c79e24d 2024-03-18T01:19:36.720 INFO:teuthology.orchestra.run.smithi029.stdout:mon.smithi029 smithi029 running (8m) 64s ago 8m 46.4M 2048M 17.2.7-684-g6716c9f3 19857db3290c 0a7fff1d6dfc 2024-03-18T01:19:36.720 INFO:teuthology.orchestra.run.smithi029.stdout:mon.smithi090 smithi090 running (5m) 71s ago 5m 38.8M 2048M 17.2.7-684-g6716c9f3 19857db3290c e23555eb8cd4 2024-03-18T01:19:36.720 INFO:teuthology.orchestra.run.smithi029.stdout:node-exporter.smithi029 smithi029 *:9100 running (7m) 64s ago 7m 15.8M - 1.5.0 0da6a335fe13 35b19f9a6014 2024-03-18T01:19:36.720 INFO:teuthology.orchestra.run.smithi029.stdout:node-exporter.smithi090 smithi090 *:9100 running (6m) 71s ago 6m 15.5M - 1.5.0 0da6a335fe13 46172a1dd8f1 2024-03-18T01:19:36.721 INFO:teuthology.orchestra.run.smithi029.stdout:osd.0 smithi029 running (5m) 64s ago 5m 46.2M 4289M 17.2.7-684-g6716c9f3 19857db3290c b34def6ff829 2024-03-18T01:19:36.721 INFO:teuthology.orchestra.run.smithi029.stdout:osd.1 smithi029 running (4m) 64s ago 4m 45.8M 4289M 17.2.7-684-g6716c9f3 19857db3290c f8915708098a 2024-03-18T01:19:36.721 INFO:teuthology.orchestra.run.smithi029.stdout:osd.2 smithi029 running (4m) 64s ago 4m 44.2M 4289M 17.2.7-684-g6716c9f3 19857db3290c cbf4647c043d 2024-03-18T01:19:36.721 INFO:teuthology.orchestra.run.smithi029.stdout:osd.3 smithi090 running (3m) 71s ago 3m 45.4M 5312M 17.2.7-684-g6716c9f3 19857db3290c dc4a91ea75c6 2024-03-18T01:19:36.721 INFO:teuthology.orchestra.run.smithi029.stdout:osd.4 smithi090 running (3m) 71s ago 3m 44.5M 5312M 17.2.7-684-g6716c9f3 19857db3290c 64754105e490 2024-03-18T01:19:36.721 INFO:teuthology.orchestra.run.smithi029.stdout:osd.5 smithi090 running (3m) 71s ago 3m 41.9M 5312M 17.2.7-684-g6716c9f3 19857db3290c 951ee2b3bc44 2024-03-18T01:19:36.721 INFO:teuthology.orchestra.run.smithi029.stdout:prometheus.smithi029 smithi029 *:9095 running (5m) 64s ago 6m 54.0M - 2.43.0 a07b618ecd1d a36de8869940 2024-03-18T01:19:36.722 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:36.720+0000 7fed27758700 1 -- 172.21.15.29:0/1244741459 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7fed0c0692a0 msgr2=0x7fed0c06b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:19:36.722 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:36.720+0000 7fed27758700 1 --2- 172.21.15.29:0/1244741459 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7fed0c0692a0 0x7fed0c06b750 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fed10009b30 tx=0x7fed10009e30 comp rx=0 tx=0).stop 2024-03-18T01:19:36.722 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:36.720+0000 7fed27758700 1 -- 172.21.15.29:0/1244741459 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fed200fbcd0 msgr2=0x7fed2010d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:19:36.722 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:36.720+0000 7fed27758700 1 --2- 172.21.15.29:0/1244741459 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fed200fbcd0 0x7fed2010d9e0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fed1c00eb40 tx=0x7fed1c00b040 comp rx=0 tx=0).stop 2024-03-18T01:19:36.722 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:36.720+0000 7fed27758700 1 -- 172.21.15.29:0/1244741459 shutdown_connections 2024-03-18T01:19:36.722 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:36.720+0000 7fed27758700 1 --2- 172.21.15.29:0/1244741459 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7fed0c0692a0 0x7fed0c06b750 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:19:36.722 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:36.720+0000 7fed27758700 1 --2- 172.21.15.29:0/1244741459 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fed200fbcd0 0x7fed2010d9e0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:19:36.722 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:36.720+0000 7fed27758700 1 --2- 172.21.15.29:0/1244741459 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fed200fcf80 0x7fed2010df20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:19:36.722 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:36.720+0000 7fed27758700 1 -- 172.21.15.29:0/1244741459 >> 172.21.15.29:0/1244741459 conn(0x7fed200f7430 msgr2=0x7fed20065ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:19:36.722 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:36.720+0000 7fed27758700 1 -- 172.21.15.29:0/1244741459 shutdown_connections 2024-03-18T01:19:36.723 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:36.721+0000 7fed27758700 1 -- 172.21.15.29:0/1244741459 wait complete. 2024-03-18T01:19:36.831 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:36.830+0000 7fa06e1f7700 1 -- 172.21.15.29:0/1382395746 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa068066280 msgr2=0x7fa0680ffa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:19:36.831 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:36.830+0000 7fa06e1f7700 1 --2- 172.21.15.29:0/1382395746 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa068066280 0x7fa0680ffa20 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fa058004940 tx=0x7fa058009e30 comp rx=0 tx=0).stop 2024-03-18T01:19:36.831 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:36.830+0000 7fa06e1f7700 1 -- 172.21.15.29:0/1382395746 shutdown_connections 2024-03-18T01:19:36.832 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:36.830+0000 7fa06e1f7700 1 --2- 172.21.15.29:0/1382395746 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa0680fff60 0x7fa068102340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:19:36.832 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:36.830+0000 7fa06e1f7700 1 --2- 172.21.15.29:0/1382395746 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa068066280 0x7fa0680ffa20 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:19:36.832 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:36.830+0000 7fa06e1f7700 1 -- 172.21.15.29:0/1382395746 >> 172.21.15.29:0/1382395746 conn(0x7fa0680f7430 msgr2=0x7fa0680f9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:19:36.832 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:36.830+0000 7fa06e1f7700 1 -- 172.21.15.29:0/1382395746 shutdown_connections 2024-03-18T01:19:36.833 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:36.830+0000 7fa06e1f7700 1 -- 172.21.15.29:0/1382395746 wait complete. 2024-03-18T01:19:36.835 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:36.830+0000 7fa06e1f7700 1 Processor -- start 2024-03-18T01:19:36.835 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:36.830+0000 7fa06e1f7700 1 -- start start 2024-03-18T01:19:36.835 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:36.831+0000 7fa06e1f7700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa068066280 0x7fa06810da80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:19:36.835 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:36.831+0000 7fa06e1f7700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa0680fff60 0x7fa06810dfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:19:36.835 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:36.831+0000 7fa06e1f7700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa06810e5c0 con 0x7fa068066280 2024-03-18T01:19:36.835 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:36.831+0000 7fa06e1f7700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa06810edc0 con 0x7fa0680fff60 2024-03-18T01:19:36.835 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:36.831+0000 7fa0677fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa068066280 0x7fa06810da80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:19:36.835 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:36.831+0000 7fa0677fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa068066280 0x7fa06810da80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:55976/0 (socket says 172.21.15.29:55976) 2024-03-18T01:19:36.836 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:36.831+0000 7fa0677fe700 1 -- 172.21.15.29:0/2795525494 learned_addr learned my addr 172.21.15.29:0/2795525494 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-18T01:19:36.836 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:36.831+0000 7fa066ffd700 1 --2- 172.21.15.29:0/2795525494 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa0680fff60 0x7fa06810dfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:19:36.836 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:36.831+0000 7fa066ffd700 1 -- 172.21.15.29:0/2795525494 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa068066280 msgr2=0x7fa06810da80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:19:36.836 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:36.831+0000 7fa066ffd700 1 --2- 172.21.15.29:0/2795525494 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa068066280 0x7fa06810da80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:19:36.836 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:36.831+0000 7fa066ffd700 1 -- 172.21.15.29:0/2795525494 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa058005040 con 0x7fa0680fff60 2024-03-18T01:19:36.836 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:36.831+0000 7fa0677fe700 1 --2- 172.21.15.29:0/2795525494 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa068066280 0x7fa06810da80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T01:19:36.836 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:36.831+0000 7fa066ffd700 1 --2- 172.21.15.29:0/2795525494 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa0680fff60 0x7fa06810dfc0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fa05c009fd0 tx=0x7fa05c00b040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:19:36.836 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:36.832+0000 7fa04ffff700 1 -- 172.21.15.29:0/2795525494 <== mon.1 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa05c004e90 con 0x7fa0680fff60 2024-03-18T01:19:36.836 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:36.832+0000 7fa06e1f7700 1 -- 172.21.15.29:0/2795525494 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa06810f040 con 0x7fa0680fff60 2024-03-18T01:19:36.836 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:36.832+0000 7fa04ffff700 1 -- 172.21.15.29:0/2795525494 <== mon.1 v2:172.21.15.90:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa05c00f040 con 0x7fa0680fff60 2024-03-18T01:19:36.836 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:36.832+0000 7fa04ffff700 1 -- 172.21.15.29:0/2795525494 <== mon.1 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa05c00ca50 con 0x7fa0680fff60 2024-03-18T01:19:36.836 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:36.832+0000 7fa06e1f7700 1 -- 172.21.15.29:0/2795525494 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa06810f580 con 0x7fa0680fff60 2024-03-18T01:19:36.836 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:36.833+0000 7fa06e1f7700 1 -- 172.21.15.29:0/2795525494 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa06805f5f0 con 0x7fa0680fff60 2024-03-18T01:19:36.836 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:36.833+0000 7fa04ffff700 1 -- 172.21.15.29:0/2795525494 <== mon.1 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7fa05c00cbb0 con 0x7fa0680fff60 2024-03-18T01:19:36.837 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:36.834+0000 7fa04ffff700 1 --2- 172.21.15.29:0/2795525494 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7fa0500692a0 0x7fa05006b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:19:36.837 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:36.834+0000 7fa04ffff700 1 -- 172.21.15.29:0/2795525494 <== mon.1 v2:172.21.15.90:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7fa05c086c20 con 0x7fa0680fff60 2024-03-18T01:19:36.837 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:36.834+0000 7fa0677fe700 1 --2- 172.21.15.29:0/2795525494 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7fa0500692a0 0x7fa05006b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:19:36.837 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:36.834+0000 7fa0677fe700 1 --2- 172.21.15.29:0/2795525494 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7fa0500692a0 0x7fa05006b750 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fa0580045a0 tx=0x7fa058009e30 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:19:36.839 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:36.838+0000 7fa04ffff700 1 -- 172.21.15.29:0/2795525494 <== mon.1 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175520 (secure 0 0 0) 0x7fa05c04ddb0 con 0x7fa0680fff60 2024-03-18T01:19:37.136 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:37.136+0000 7fa06e1f7700 1 -- 172.21.15.29:0/2795525494 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fa06810f7f0 con 0x7fa0680fff60 2024-03-18T01:19:37.137 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:37.137+0000 7fa04ffff700 1 -- 172.21.15.29:0/2795525494 <== mon.1 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+638 (secure 0 0 0) 0x7fa05c0569a0 con 0x7fa0680fff60 2024-03-18T01:19:37.137 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-03-18T01:19:37.137 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": { 2024-03-18T01:19:37.137 INFO:teuthology.orchestra.run.smithi029.stdout: "ceph version 17.2.7-684-g6716c9f3 (6716c9f3976cf9fc2fd546e9f1182c2d4421b451) quincy (stable)": 2 2024-03-18T01:19:37.137 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-18T01:19:37.138 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": { 2024-03-18T01:19:37.138 INFO:teuthology.orchestra.run.smithi029.stdout: "ceph version 17.2.7-684-g6716c9f3 (6716c9f3976cf9fc2fd546e9f1182c2d4421b451) quincy (stable)": 2 2024-03-18T01:19:37.138 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-18T01:19:37.138 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": { 2024-03-18T01:19:37.138 INFO:teuthology.orchestra.run.smithi029.stdout: "ceph version 17.2.7-684-g6716c9f3 (6716c9f3976cf9fc2fd546e9f1182c2d4421b451) quincy (stable)": 6 2024-03-18T01:19:37.138 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-18T01:19:37.138 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": { 2024-03-18T01:19:37.138 INFO:teuthology.orchestra.run.smithi029.stdout: "ceph version 17.2.7-684-g6716c9f3 (6716c9f3976cf9fc2fd546e9f1182c2d4421b451) quincy (stable)": 4 2024-03-18T01:19:37.138 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-18T01:19:37.138 INFO:teuthology.orchestra.run.smithi029.stdout: "overall": { 2024-03-18T01:19:37.138 INFO:teuthology.orchestra.run.smithi029.stdout: "ceph version 17.2.7-684-g6716c9f3 (6716c9f3976cf9fc2fd546e9f1182c2d4421b451) quincy (stable)": 14 2024-03-18T01:19:37.138 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-18T01:19:37.138 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-03-18T01:19:37.140 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:37.139+0000 7fa06e1f7700 1 -- 172.21.15.29:0/2795525494 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7fa0500692a0 msgr2=0x7fa05006b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:19:37.140 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:37.139+0000 7fa06e1f7700 1 --2- 172.21.15.29:0/2795525494 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7fa0500692a0 0x7fa05006b750 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fa0580045a0 tx=0x7fa058009e30 comp rx=0 tx=0).stop 2024-03-18T01:19:37.140 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:37.139+0000 7fa06e1f7700 1 -- 172.21.15.29:0/2795525494 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa0680fff60 msgr2=0x7fa06810dfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:19:37.140 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:37.139+0000 7fa06e1f7700 1 --2- 172.21.15.29:0/2795525494 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa0680fff60 0x7fa06810dfc0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fa05c009fd0 tx=0x7fa05c00b040 comp rx=0 tx=0).stop 2024-03-18T01:19:37.140 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:37.139+0000 7fa06e1f7700 1 -- 172.21.15.29:0/2795525494 shutdown_connections 2024-03-18T01:19:37.140 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:37.139+0000 7fa06e1f7700 1 --2- 172.21.15.29:0/2795525494 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7fa0500692a0 0x7fa05006b750 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:19:37.140 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:37.139+0000 7fa06e1f7700 1 --2- 172.21.15.29:0/2795525494 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa068066280 0x7fa06810da80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:19:37.140 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:37.139+0000 7fa06e1f7700 1 --2- 172.21.15.29:0/2795525494 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa0680fff60 0x7fa06810dfc0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:19:37.140 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:37.139+0000 7fa06e1f7700 1 -- 172.21.15.29:0/2795525494 >> 172.21.15.29:0/2795525494 conn(0x7fa0680f7430 msgr2=0x7fa0680f7850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:19:37.140 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:37.139+0000 7fa06e1f7700 1 -- 172.21.15.29:0/2795525494 shutdown_connections 2024-03-18T01:19:37.140 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:37.140+0000 7fa06e1f7700 1 -- 172.21.15.29:0/2795525494 wait complete. 2024-03-18T01:19:37.256 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:37.255+0000 7ff561ffd700 1 -- 172.21.15.29:0/3203583590 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff55c0fcf80 msgr2=0x7ff55c0fd3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:19:37.256 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:37.255+0000 7ff561ffd700 1 --2- 172.21.15.29:0/3203583590 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff55c0fcf80 0x7ff55c0fd3f0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7ff550004940 tx=0x7ff550009e30 comp rx=0 tx=0).stop 2024-03-18T01:19:37.257 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:37.255+0000 7ff561ffd700 1 -- 172.21.15.29:0/3203583590 shutdown_connections 2024-03-18T01:19:37.257 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:37.255+0000 7ff561ffd700 1 --2- 172.21.15.29:0/3203583590 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff55c0fcf80 0x7ff55c0fd3f0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:19:37.257 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:37.255+0000 7ff561ffd700 1 --2- 172.21.15.29:0/3203583590 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff55c0fbcd0 0x7ff55c0fc0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:19:37.257 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:37.255+0000 7ff561ffd700 1 -- 172.21.15.29:0/3203583590 >> 172.21.15.29:0/3203583590 conn(0x7ff55c0f7430 msgr2=0x7ff55c0f9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:19:37.257 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:37.256+0000 7ff561ffd700 1 -- 172.21.15.29:0/3203583590 shutdown_connections 2024-03-18T01:19:37.257 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:37.256+0000 7ff561ffd700 1 -- 172.21.15.29:0/3203583590 wait complete. 2024-03-18T01:19:37.257 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:37.256+0000 7ff561ffd700 1 Processor -- start 2024-03-18T01:19:37.257 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:37.256+0000 7ff561ffd700 1 -- start start 2024-03-18T01:19:37.258 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:37.256+0000 7ff561ffd700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff55c0fbcd0 0x7ff55c0ffda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:19:37.259 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:37.256+0000 7ff561ffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff55c0fcf80 0x7ff55c0fe2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:19:37.259 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:37.257+0000 7ff561ffd700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff55c0fe850 con 0x7ff55c0fcf80 2024-03-18T01:19:37.259 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:37.257+0000 7ff561ffd700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff55c0fe990 con 0x7ff55c0fbcd0 2024-03-18T01:19:37.259 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:37.257+0000 7ff55b7fe700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff55c0fbcd0 0x7ff55c0ffda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:19:37.259 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:37.257+0000 7ff55affd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff55c0fcf80 0x7ff55c0fe2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:19:37.259 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:37.257+0000 7ff55b7fe700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff55c0fbcd0 0x7ff55c0ffda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.29:59472/0 (socket says 172.21.15.29:59472) 2024-03-18T01:19:37.259 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:37.257+0000 7ff55b7fe700 1 -- 172.21.15.29:0/406863511 learned_addr learned my addr 172.21.15.29:0/406863511 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-18T01:19:37.259 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:37.257+0000 7ff55b7fe700 1 -- 172.21.15.29:0/406863511 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff55c0fcf80 msgr2=0x7ff55c0fe2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:19:37.259 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:37.257+0000 7ff55b7fe700 1 --2- 172.21.15.29:0/406863511 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff55c0fcf80 0x7ff55c0fe2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:19:37.259 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:37.257+0000 7ff55b7fe700 1 -- 172.21.15.29:0/406863511 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff550005040 con 0x7ff55c0fbcd0 2024-03-18T01:19:37.259 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:37.257+0000 7ff55affd700 1 --2- 172.21.15.29:0/406863511 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff55c0fcf80 0x7ff55c0fe2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T01:19:37.259 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:37.257+0000 7ff55b7fe700 1 --2- 172.21.15.29:0/406863511 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff55c0fbcd0 0x7ff55c0ffda0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7ff54c00eb40 tx=0x7ff54c00b040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:19:37.259 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:37.257+0000 7ff543fff700 1 -- 172.21.15.29:0/406863511 <== mon.1 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff54c007cc0 con 0x7ff55c0fbcd0 2024-03-18T01:19:37.259 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:37.257+0000 7ff561ffd700 1 -- 172.21.15.29:0/406863511 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff55c0fec10 con 0x7ff55c0fbcd0 2024-03-18T01:19:37.260 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:37.257+0000 7ff543fff700 1 -- 172.21.15.29:0/406863511 <== mon.1 v2:172.21.15.90:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff54c00f040 con 0x7ff55c0fbcd0 2024-03-18T01:19:37.260 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:37.258+0000 7ff543fff700 1 -- 172.21.15.29:0/406863511 <== mon.1 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff54c00ca10 con 0x7ff55c0fbcd0 2024-03-18T01:19:37.260 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:37.258+0000 7ff561ffd700 1 -- 172.21.15.29:0/406863511 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff55c0ff120 con 0x7ff55c0fbcd0 2024-03-18T01:19:37.261 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:37.259+0000 7ff561ffd700 1 -- 172.21.15.29:0/406863511 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff55c101b10 con 0x7ff55c0fbcd0 2024-03-18T01:19:37.261 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:37.259+0000 7ff543fff700 1 -- 172.21.15.29:0/406863511 <== mon.1 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7ff54c00cc80 con 0x7ff55c0fbcd0 2024-03-18T01:19:37.262 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:37.260+0000 7ff543fff700 1 --2- 172.21.15.29:0/406863511 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7ff5440692a0 0x7ff54406b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:19:37.262 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:37.260+0000 7ff543fff700 1 -- 172.21.15.29:0/406863511 <== mon.1 v2:172.21.15.90:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7ff54c086dd0 con 0x7ff55c0fbcd0 2024-03-18T01:19:37.262 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:37.260+0000 7ff55affd700 1 --2- 172.21.15.29:0/406863511 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7ff5440692a0 0x7ff54406b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:19:37.262 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:37.260+0000 7ff55affd700 1 --2- 172.21.15.29:0/406863511 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7ff5440692a0 0x7ff54406b750 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7ff550009b30 tx=0x7ff550009e30 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:19:37.264 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:37.264+0000 7ff543fff700 1 -- 172.21.15.29:0/406863511 <== mon.1 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175520 (secure 0 0 0) 0x7ff54c04ddb0 con 0x7ff55c0fbcd0 2024-03-18T01:19:37.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:19:37 smithi090 ceph-mon[36989]: Upgrade: Target is version 19.0.0-2162-g3d78f5ba (unknown) 2024-03-18T01:19:37.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:19:37 smithi090 ceph-mon[36989]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, digests ['quay.ceph.io/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513'] 2024-03-18T01:19:37.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:19:37 smithi090 ceph-mon[36989]: Upgrade: Need to upgrade myself (mgr.smithi029.qviwhl) 2024-03-18T01:19:37.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:19:37 smithi090 ceph-mon[36989]: from='client.24379 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T01:19:37.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:19:37 smithi090 ceph-mon[36989]: Upgrade: Pulling quay.ceph.io/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 on smithi090 2024-03-18T01:19:37.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:19:37 smithi090 ceph-mon[36989]: from='client.24383 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T01:19:37.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:19:37 smithi090 ceph-mon[36989]: from='client.? 172.21.15.29:0/2795525494' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-18T01:19:37.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:19:37 smithi029 ceph-mon[29331]: Upgrade: Target is version 19.0.0-2162-g3d78f5ba (unknown) 2024-03-18T01:19:37.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:19:37 smithi029 ceph-mon[29331]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, digests ['quay.ceph.io/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513'] 2024-03-18T01:19:37.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:19:37 smithi029 ceph-mon[29331]: Upgrade: Need to upgrade myself (mgr.smithi029.qviwhl) 2024-03-18T01:19:37.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:19:37 smithi029 ceph-mon[29331]: from='client.24379 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T01:19:37.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:19:37 smithi029 ceph-mon[29331]: Upgrade: Pulling quay.ceph.io/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 on smithi090 2024-03-18T01:19:37.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:19:37 smithi029 ceph-mon[29331]: from='client.24383 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T01:19:37.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:19:37 smithi029 ceph-mon[29331]: from='client.? 172.21.15.29:0/2795525494' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-18T01:19:37.572 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:37.572+0000 7ff561ffd700 1 -- 172.21.15.29:0/406863511 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7ff55c02b2a0 con 0x7ff55c0fbcd0 2024-03-18T01:19:37.572 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:37.572+0000 7ff543fff700 1 -- 172.21.15.29:0/406863511 <== mon.1 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 21 v21) v1 ==== 76+0+1842 (secure 0 0 0) 0x7ff54c0530e0 con 0x7ff55c0fbcd0 2024-03-18T01:19:37.573 INFO:teuthology.orchestra.run.smithi029.stdout:e21 2024-03-18T01:19:37.573 INFO:teuthology.orchestra.run.smithi029.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-18T01:19:37.573 INFO:teuthology.orchestra.run.smithi029.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-18T01:19:37.573 INFO:teuthology.orchestra.run.smithi029.stdout:legacy client fscid: 1 2024-03-18T01:19:37.573 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-18T01:19:37.573 INFO:teuthology.orchestra.run.smithi029.stdout:Filesystem 'cephfs' (1) 2024-03-18T01:19:37.573 INFO:teuthology.orchestra.run.smithi029.stdout:fs_name cephfs 2024-03-18T01:19:37.573 INFO:teuthology.orchestra.run.smithi029.stdout:epoch 21 2024-03-18T01:19:37.573 INFO:teuthology.orchestra.run.smithi029.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-03-18T01:19:37.573 INFO:teuthology.orchestra.run.smithi029.stdout:created 2024-03-18T01:18:03.228199+0000 2024-03-18T01:19:37.573 INFO:teuthology.orchestra.run.smithi029.stdout:modified 2024-03-18T01:19:33.153090+0000 2024-03-18T01:19:37.573 INFO:teuthology.orchestra.run.smithi029.stdout:tableserver 0 2024-03-18T01:19:37.574 INFO:teuthology.orchestra.run.smithi029.stdout:root 0 2024-03-18T01:19:37.574 INFO:teuthology.orchestra.run.smithi029.stdout:session_timeout 60 2024-03-18T01:19:37.574 INFO:teuthology.orchestra.run.smithi029.stdout:session_autoclose 300 2024-03-18T01:19:37.574 INFO:teuthology.orchestra.run.smithi029.stdout:max_file_size 1099511627776 2024-03-18T01:19:37.574 INFO:teuthology.orchestra.run.smithi029.stdout:required_client_features {} 2024-03-18T01:19:37.574 INFO:teuthology.orchestra.run.smithi029.stdout:last_failure 0 2024-03-18T01:19:37.574 INFO:teuthology.orchestra.run.smithi029.stdout:last_failure_osd_epoch 46 2024-03-18T01:19:37.574 INFO:teuthology.orchestra.run.smithi029.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-18T01:19:37.574 INFO:teuthology.orchestra.run.smithi029.stdout:max_mds 2 2024-03-18T01:19:37.574 INFO:teuthology.orchestra.run.smithi029.stdout:in 0,1 2024-03-18T01:19:37.574 INFO:teuthology.orchestra.run.smithi029.stdout:up {0=14480,1=24281} 2024-03-18T01:19:37.574 INFO:teuthology.orchestra.run.smithi029.stdout:failed 2024-03-18T01:19:37.574 INFO:teuthology.orchestra.run.smithi029.stdout:damaged 2024-03-18T01:19:37.574 INFO:teuthology.orchestra.run.smithi029.stdout:stopped 2024-03-18T01:19:37.574 INFO:teuthology.orchestra.run.smithi029.stdout:data_pools [3] 2024-03-18T01:19:37.575 INFO:teuthology.orchestra.run.smithi029.stdout:metadata_pool 2 2024-03-18T01:19:37.575 INFO:teuthology.orchestra.run.smithi029.stdout:inline_data enabled 2024-03-18T01:19:37.575 INFO:teuthology.orchestra.run.smithi029.stdout:balancer 2024-03-18T01:19:37.575 INFO:teuthology.orchestra.run.smithi029.stdout:standby_count_wanted 1 2024-03-18T01:19:37.575 INFO:teuthology.orchestra.run.smithi029.stdout:[mds.cephfs.smithi029.brxqeh{0:14480} state up:active seq 6 export targets 1 join_fscid=1 addr [v2:172.21.15.29:6828/1299751620,v1:172.21.15.29:6829/1299751620] compat {c=[1],r=[1],i=[7ff]}] 2024-03-18T01:19:37.575 INFO:teuthology.orchestra.run.smithi029.stdout:[mds.cephfs.smithi029.xuxvlr{0:14492} state up:standby-replay seq 1 join_fscid=1 addr [v2:172.21.15.29:6826/890079928,v1:172.21.15.29:6827/890079928] compat {c=[1],r=[1],i=[7ff]}] 2024-03-18T01:19:37.575 INFO:teuthology.orchestra.run.smithi029.stdout:[mds.cephfs.smithi090.bohder{1:24281} state up:active seq 4 join_fscid=1 addr [v2:172.21.15.90:6824/1998449484,v1:172.21.15.90:6825/1998449484] compat {c=[1],r=[1],i=[7ff]}] 2024-03-18T01:19:37.575 INFO:teuthology.orchestra.run.smithi029.stdout:[mds.cephfs.smithi090.trsnxz{1:24297} state up:standby-replay seq 2 join_fscid=1 addr [v2:172.21.15.90:6826/439410941,v1:172.21.15.90:6827/439410941] compat {c=[1],r=[1],i=[7ff]}] 2024-03-18T01:19:37.575 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-18T01:19:37.575 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-18T01:19:37.578 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:37.575+0000 7ff561ffd700 1 -- 172.21.15.29:0/406863511 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7ff5440692a0 msgr2=0x7ff54406b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:19:37.578 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:37.575+0000 7ff561ffd700 1 --2- 172.21.15.29:0/406863511 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7ff5440692a0 0x7ff54406b750 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7ff550009b30 tx=0x7ff550009e30 comp rx=0 tx=0).stop 2024-03-18T01:19:37.578 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:37.575+0000 7ff561ffd700 1 -- 172.21.15.29:0/406863511 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff55c0fbcd0 msgr2=0x7ff55c0ffda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:19:37.578 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:37.575+0000 7ff561ffd700 1 --2- 172.21.15.29:0/406863511 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff55c0fbcd0 0x7ff55c0ffda0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7ff54c00eb40 tx=0x7ff54c00b040 comp rx=0 tx=0).stop 2024-03-18T01:19:37.578 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:37.575+0000 7ff561ffd700 1 -- 172.21.15.29:0/406863511 shutdown_connections 2024-03-18T01:19:37.578 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:37.575+0000 7ff561ffd700 1 --2- 172.21.15.29:0/406863511 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7ff5440692a0 0x7ff54406b750 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:19:37.578 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:37.575+0000 7ff561ffd700 1 --2- 172.21.15.29:0/406863511 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff55c0fbcd0 0x7ff55c0ffda0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:19:37.578 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:37.575+0000 7ff561ffd700 1 --2- 172.21.15.29:0/406863511 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff55c0fcf80 0x7ff55c0fe2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:19:37.578 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:37.575+0000 7ff561ffd700 1 -- 172.21.15.29:0/406863511 >> 172.21.15.29:0/406863511 conn(0x7ff55c0f7430 msgr2=0x7ff55c065ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:19:37.578 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:37.575+0000 7ff561ffd700 1 -- 172.21.15.29:0/406863511 shutdown_connections 2024-03-18T01:19:37.579 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:37.575+0000 7ff561ffd700 1 -- 172.21.15.29:0/406863511 wait complete. 2024-03-18T01:19:37.579 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 21 2024-03-18T01:19:37.689 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:37.686+0000 7f45efff3700 1 -- 172.21.15.29:0/4267942019 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f45e80662e0 msgr2=0x7f45e80ff9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:19:37.689 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:37.686+0000 7f45efff3700 1 --2- 172.21.15.29:0/4267942019 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f45e80662e0 0x7f45e80ff9c0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f45e4004940 tx=0x7f45e4009e30 comp rx=0 tx=0).stop 2024-03-18T01:19:37.689 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:37.686+0000 7f45efff3700 1 -- 172.21.15.29:0/4267942019 shutdown_connections 2024-03-18T01:19:37.689 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:37.686+0000 7f45efff3700 1 --2- 172.21.15.29:0/4267942019 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f45e80fff00 0x7f45e8102320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:19:37.689 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:37.686+0000 7f45efff3700 1 --2- 172.21.15.29:0/4267942019 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f45e80662e0 0x7f45e80ff9c0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:19:37.689 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:37.686+0000 7f45efff3700 1 -- 172.21.15.29:0/4267942019 >> 172.21.15.29:0/4267942019 conn(0x7f45e80f7430 msgr2=0x7f45e80f9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:19:37.689 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:37.686+0000 7f45efff3700 1 -- 172.21.15.29:0/4267942019 shutdown_connections 2024-03-18T01:19:37.689 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:37.686+0000 7f45efff3700 1 -- 172.21.15.29:0/4267942019 wait complete. 2024-03-18T01:19:37.689 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:37.687+0000 7f45efff3700 1 Processor -- start 2024-03-18T01:19:37.689 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:37.687+0000 7f45efff3700 1 -- start start 2024-03-18T01:19:37.690 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:37.687+0000 7f45efff3700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f45e80662e0 0x7f45e810da30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:19:37.690 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:37.687+0000 7f45efff3700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f45e80fff00 0x7f45e810df70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:19:37.690 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:37.687+0000 7f45efff3700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45e810e570 con 0x7f45e80fff00 2024-03-18T01:19:37.690 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:37.687+0000 7f45efff3700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45e810edc0 con 0x7f45e80662e0 2024-03-18T01:19:37.690 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:37.687+0000 7f45edd8f700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f45e80662e0 0x7f45e810da30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:19:37.690 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:37.687+0000 7f45ed58e700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f45e80fff00 0x7f45e810df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:19:37.690 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:37.687+0000 7f45ed58e700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f45e80fff00 0x7f45e810df70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:55998/0 (socket says 172.21.15.29:55998) 2024-03-18T01:19:37.690 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:37.687+0000 7f45ed58e700 1 -- 172.21.15.29:0/138635282 learned_addr learned my addr 172.21.15.29:0/138635282 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-18T01:19:37.690 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:37.688+0000 7f45ed58e700 1 -- 172.21.15.29:0/138635282 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f45e80662e0 msgr2=0x7f45e810da30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:19:37.690 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:37.688+0000 7f45ed58e700 1 --2- 172.21.15.29:0/138635282 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f45e80662e0 0x7f45e810da30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:19:37.690 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:37.688+0000 7f45ed58e700 1 -- 172.21.15.29:0/138635282 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45e4005040 con 0x7f45e80fff00 2024-03-18T01:19:37.691 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:37.688+0000 7f45ed58e700 1 --2- 172.21.15.29:0/138635282 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f45e80fff00 0x7f45e810df70 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f45d8002a50 tx=0x7f45d80046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:19:37.691 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:37.688+0000 7f45de7fc700 1 -- 172.21.15.29:0/138635282 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f45d8007cc0 con 0x7f45e80fff00 2024-03-18T01:19:37.691 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:37.688+0000 7f45de7fc700 1 -- 172.21.15.29:0/138635282 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f45d800f040 con 0x7f45e80fff00 2024-03-18T01:19:37.691 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:37.688+0000 7f45efff3700 1 -- 172.21.15.29:0/138635282 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45e810f0a0 con 0x7f45e80fff00 2024-03-18T01:19:37.691 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:37.688+0000 7f45de7fc700 1 -- 172.21.15.29:0/138635282 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f45d800da30 con 0x7f45e80fff00 2024-03-18T01:19:37.691 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:37.688+0000 7f45efff3700 1 -- 172.21.15.29:0/138635282 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45e810f5e0 con 0x7f45e80fff00 2024-03-18T01:19:37.693 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:37.689+0000 7f45efff3700 1 -- 172.21.15.29:0/138635282 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f45e805f5f0 con 0x7f45e80fff00 2024-03-18T01:19:37.693 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:37.690+0000 7f45de7fc700 1 -- 172.21.15.29:0/138635282 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f45d800db90 con 0x7f45e80fff00 2024-03-18T01:19:37.693 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:37.690+0000 7f45de7fc700 1 --2- 172.21.15.29:0/138635282 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f45d40692a0 0x7f45d406b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:19:37.694 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:37.690+0000 7f45de7fc700 1 -- 172.21.15.29:0/138635282 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f45d8086eb0 con 0x7f45e80fff00 2024-03-18T01:19:37.694 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:37.690+0000 7f45edd8f700 1 --2- 172.21.15.29:0/138635282 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f45d40692a0 0x7f45d406b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:19:37.694 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:37.691+0000 7f45edd8f700 1 --2- 172.21.15.29:0/138635282 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f45d40692a0 0x7f45d406b750 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f45e400b970 tx=0x7f45e4009e30 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:19:37.695 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:37.695+0000 7f45de7fc700 1 -- 172.21.15.29:0/138635282 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175520 (secure 0 0 0) 0x7f45d8056c20 con 0x7f45e80fff00 2024-03-18T01:19:37.971 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:37.970+0000 7f45efff3700 1 -- 172.21.15.29:0/138635282 --> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f45e810f9c0 con 0x7f45d40692a0 2024-03-18T01:19:37.972 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:37.972+0000 7f45de7fc700 1 -- 172.21.15.29:0/138635282 <== mgr.14208 v2:172.21.15.29:6800/1641454714 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+447 (secure 0 0 0) 0x7f45e810f9c0 con 0x7f45d40692a0 2024-03-18T01:19:37.972 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-03-18T01:19:37.972 INFO:teuthology.orchestra.run.smithi029.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513", 2024-03-18T01:19:37.972 INFO:teuthology.orchestra.run.smithi029.stdout: "in_progress": true, 2024-03-18T01:19:37.972 INFO:teuthology.orchestra.run.smithi029.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-18T01:19:37.972 INFO:teuthology.orchestra.run.smithi029.stdout: "services_complete": [], 2024-03-18T01:19:37.973 INFO:teuthology.orchestra.run.smithi029.stdout: "progress": "0/21 daemons upgraded", 2024-03-18T01:19:37.973 INFO:teuthology.orchestra.run.smithi029.stdout: "message": "Pulling quay.ceph.io/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 image on host smithi090", 2024-03-18T01:19:37.973 INFO:teuthology.orchestra.run.smithi029.stdout: "is_paused": false 2024-03-18T01:19:37.973 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-03-18T01:19:37.974 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:37.974+0000 7f45efff3700 1 -- 172.21.15.29:0/138635282 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f45d40692a0 msgr2=0x7f45d406b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:19:37.974 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:37.974+0000 7f45efff3700 1 --2- 172.21.15.29:0/138635282 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f45d40692a0 0x7f45d406b750 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f45e400b970 tx=0x7f45e4009e30 comp rx=0 tx=0).stop 2024-03-18T01:19:37.974 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:37.974+0000 7f45efff3700 1 -- 172.21.15.29:0/138635282 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f45e80fff00 msgr2=0x7f45e810df70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:19:37.974 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:37.974+0000 7f45efff3700 1 --2- 172.21.15.29:0/138635282 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f45e80fff00 0x7f45e810df70 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f45d8002a50 tx=0x7f45d80046b0 comp rx=0 tx=0).stop 2024-03-18T01:19:37.975 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:37.974+0000 7f45efff3700 1 -- 172.21.15.29:0/138635282 shutdown_connections 2024-03-18T01:19:37.975 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:37.974+0000 7f45efff3700 1 --2- 172.21.15.29:0/138635282 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7f45d40692a0 0x7f45d406b750 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:19:37.975 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:37.974+0000 7f45efff3700 1 --2- 172.21.15.29:0/138635282 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f45e80662e0 0x7f45e810da30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:19:37.975 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:37.974+0000 7f45efff3700 1 --2- 172.21.15.29:0/138635282 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f45e80fff00 0x7f45e810df70 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:19:37.975 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:37.974+0000 7f45efff3700 1 -- 172.21.15.29:0/138635282 >> 172.21.15.29:0/138635282 conn(0x7f45e80f7430 msgr2=0x7f45e80f7850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:19:37.975 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:37.974+0000 7f45efff3700 1 -- 172.21.15.29:0/138635282 shutdown_connections 2024-03-18T01:19:37.975 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:37.974+0000 7f45efff3700 1 -- 172.21.15.29:0/138635282 wait complete. 2024-03-18T01:19:38.090 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:38.088+0000 7fc191b62700 1 -- 172.21.15.29:0/1996093275 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc18c0662e0 msgr2=0x7fc18c0ff9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:19:38.090 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:38.088+0000 7fc191b62700 1 --2- 172.21.15.29:0/1996093275 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc18c0662e0 0x7fc18c0ff9c0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fc17c004940 tx=0x7fc17c009e30 comp rx=0 tx=0).stop 2024-03-18T01:19:38.091 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:38.088+0000 7fc191b62700 1 -- 172.21.15.29:0/1996093275 shutdown_connections 2024-03-18T01:19:38.091 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:38.088+0000 7fc191b62700 1 --2- 172.21.15.29:0/1996093275 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc18c0fff00 0x7fc18c102320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:19:38.091 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:38.088+0000 7fc191b62700 1 --2- 172.21.15.29:0/1996093275 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc18c0662e0 0x7fc18c0ff9c0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:19:38.091 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:38.088+0000 7fc191b62700 1 -- 172.21.15.29:0/1996093275 >> 172.21.15.29:0/1996093275 conn(0x7fc18c0f7430 msgr2=0x7fc18c0f9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:19:38.091 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:38.088+0000 7fc191b62700 1 -- 172.21.15.29:0/1996093275 shutdown_connections 2024-03-18T01:19:38.091 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:38.088+0000 7fc191b62700 1 -- 172.21.15.29:0/1996093275 wait complete. 2024-03-18T01:19:38.091 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:38.088+0000 7fc191b62700 1 Processor -- start 2024-03-18T01:19:38.091 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:38.089+0000 7fc191b62700 1 -- start start 2024-03-18T01:19:38.091 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:38.089+0000 7fc191b62700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc18c0662e0 0x7fc18c0fd3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:19:38.091 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:38.089+0000 7fc191b62700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc18c0fff00 0x7fc18c0fb8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:19:38.091 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:38.089+0000 7fc191b62700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc18c0fbe50 con 0x7fc18c0fff00 2024-03-18T01:19:38.091 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:38.089+0000 7fc191b62700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc18c0fbf90 con 0x7fc18c0662e0 2024-03-18T01:19:38.091 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:38.089+0000 7fc18affd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc18c0fff00 0x7fc18c0fb8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:19:38.091 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:38.089+0000 7fc18b7fe700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc18c0662e0 0x7fc18c0fd3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:19:38.092 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:38.089+0000 7fc18affd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc18c0fff00 0x7fc18c0fb8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56014/0 (socket says 172.21.15.29:56014) 2024-03-18T01:19:38.092 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:38.089+0000 7fc18b7fe700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc18c0662e0 0x7fc18c0fd3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.29:59510/0 (socket says 172.21.15.29:59510) 2024-03-18T01:19:38.092 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:38.089+0000 7fc18affd700 1 -- 172.21.15.29:0/1404630693 learned_addr learned my addr 172.21.15.29:0/1404630693 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-18T01:19:38.092 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:38.089+0000 7fc18b7fe700 1 -- 172.21.15.29:0/1404630693 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc18c0fff00 msgr2=0x7fc18c0fb8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:19:38.092 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:38.089+0000 7fc18b7fe700 1 --2- 172.21.15.29:0/1404630693 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc18c0fff00 0x7fc18c0fb8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:19:38.092 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:38.089+0000 7fc18b7fe700 1 -- 172.21.15.29:0/1404630693 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc17c005040 con 0x7fc18c0662e0 2024-03-18T01:19:38.092 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:38.089+0000 7fc18b7fe700 1 --2- 172.21.15.29:0/1404630693 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc18c0662e0 0x7fc18c0fd3a0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fc17c000c00 tx=0x7fc17c009e30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:19:38.092 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:38.090+0000 7fc173fff700 1 -- 172.21.15.29:0/1404630693 <== mon.1 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc17c00e070 con 0x7fc18c0662e0 2024-03-18T01:19:38.092 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:38.090+0000 7fc191b62700 1 -- 172.21.15.29:0/1404630693 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc18c0fc210 con 0x7fc18c0662e0 2024-03-18T01:19:38.092 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:38.090+0000 7fc191b62700 1 -- 172.21.15.29:0/1404630693 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc18c115a30 con 0x7fc18c0662e0 2024-03-18T01:19:38.092 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:38.090+0000 7fc173fff700 1 -- 172.21.15.29:0/1404630693 <== mon.1 v2:172.21.15.90:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc17c013de0 con 0x7fc18c0662e0 2024-03-18T01:19:38.092 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:38.090+0000 7fc173fff700 1 -- 172.21.15.29:0/1404630693 <== mon.1 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc17c01cde0 con 0x7fc18c0662e0 2024-03-18T01:19:38.093 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:38.091+0000 7fc191b62700 1 -- 172.21.15.29:0/1404630693 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1780052e0 con 0x7fc18c0662e0 2024-03-18T01:19:38.093 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:38.092+0000 7fc173fff700 1 -- 172.21.15.29:0/1404630693 <== mon.1 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7fc17c0263f0 con 0x7fc18c0662e0 2024-03-18T01:19:38.094 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:38.092+0000 7fc173fff700 1 --2- 172.21.15.29:0/1404630693 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7fc1740692b0 0x7fc17406b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:19:38.094 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:38.092+0000 7fc173fff700 1 -- 172.21.15.29:0/1404630693 <== mon.1 v2:172.21.15.90:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7fc17c0878c0 con 0x7fc18c0662e0 2024-03-18T01:19:38.094 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:38.092+0000 7fc18affd700 1 --2- 172.21.15.29:0/1404630693 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7fc1740692b0 0x7fc17406b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:19:38.094 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:38.093+0000 7fc18affd700 1 --2- 172.21.15.29:0/1404630693 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7fc1740692b0 0x7fc17406b760 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fc180008fd0 tx=0x7fc18000b040 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:19:38.096 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:38.096+0000 7fc173fff700 1 -- 172.21.15.29:0/1404630693 <== mon.1 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175520 (secure 0 0 0) 0x7fc17c053cf0 con 0x7fc18c0662e0 2024-03-18T01:19:38.319 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:19:38 smithi029 ceph-mon[29331]: from='client.14596 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T01:19:38.319 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:19:38 smithi029 ceph-mon[29331]: pgmap v227: 65 pgs: 65 active+clean; 48 MiB data, 1.9 GiB used, 535 GiB / 536 GiB avail; 1.2 MiB/s rd, 4.2 MiB/s wr, 360 op/s 2024-03-18T01:19:38.319 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:19:38 smithi029 ceph-mon[29331]: from='client.? 172.21.15.29:0/406863511' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-18T01:19:38.427 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:19:38 smithi090 ceph-mon[36989]: from='client.14596 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T01:19:38.427 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:19:38 smithi090 ceph-mon[36989]: pgmap v227: 65 pgs: 65 active+clean; 48 MiB data, 1.9 GiB used, 535 GiB / 536 GiB avail; 1.2 MiB/s rd, 4.2 MiB/s wr, 360 op/s 2024-03-18T01:19:38.427 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:19:38 smithi090 ceph-mon[36989]: from='client.? 172.21.15.29:0/406863511' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-18T01:19:38.467 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:38.466+0000 7fc191b62700 1 -- 172.21.15.29:0/1404630693 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fc178005590 con 0x7fc18c0662e0 2024-03-18T01:19:38.467 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:38.467+0000 7fc173fff700 1 -- 172.21.15.29:0/1404630693 <== mon.1 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+201 (secure 0 0 0) 0x7fc17c022070 con 0x7fc18c0662e0 2024-03-18T01:19:38.467 INFO:teuthology.orchestra.run.smithi029.stdout:HEALTH_WARN 1 filesystem with deprecated feature inline_data 2024-03-18T01:19:38.467 INFO:teuthology.orchestra.run.smithi029.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-03-18T01:19:38.467 INFO:teuthology.orchestra.run.smithi029.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-03-18T01:19:38.469 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:38.469+0000 7fc191b62700 1 -- 172.21.15.29:0/1404630693 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7fc1740692b0 msgr2=0x7fc17406b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:19:38.469 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:38.469+0000 7fc191b62700 1 --2- 172.21.15.29:0/1404630693 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7fc1740692b0 0x7fc17406b760 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fc180008fd0 tx=0x7fc18000b040 comp rx=0 tx=0).stop 2024-03-18T01:19:38.469 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:38.469+0000 7fc191b62700 1 -- 172.21.15.29:0/1404630693 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc18c0662e0 msgr2=0x7fc18c0fd3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:19:38.469 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:38.469+0000 7fc191b62700 1 --2- 172.21.15.29:0/1404630693 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc18c0662e0 0x7fc18c0fd3a0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fc17c000c00 tx=0x7fc17c009e30 comp rx=0 tx=0).stop 2024-03-18T01:19:38.469 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:38.469+0000 7fc191b62700 1 -- 172.21.15.29:0/1404630693 shutdown_connections 2024-03-18T01:19:38.469 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:38.469+0000 7fc191b62700 1 --2- 172.21.15.29:0/1404630693 >> [v2:172.21.15.29:6800/1641454714,v1:172.21.15.29:6801/1641454714] conn(0x7fc1740692b0 0x7fc17406b760 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:19:38.469 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:38.469+0000 7fc191b62700 1 --2- 172.21.15.29:0/1404630693 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc18c0662e0 0x7fc18c0fd3a0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:19:38.470 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:38.469+0000 7fc191b62700 1 --2- 172.21.15.29:0/1404630693 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc18c0fff00 0x7fc18c0fb8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:19:38.470 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:38.469+0000 7fc191b62700 1 -- 172.21.15.29:0/1404630693 >> 172.21.15.29:0/1404630693 conn(0x7fc18c0f7430 msgr2=0x7fc18c0f7850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:19:38.470 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:38.469+0000 7fc191b62700 1 -- 172.21.15.29:0/1404630693 shutdown_connections 2024-03-18T01:19:38.470 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:19:38.469+0000 7fc191b62700 1 -- 172.21.15.29:0/1404630693 wait complete. 2024-03-18T01:19:39.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:19:39 smithi090 ceph-mon[36989]: from='client.14608 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T01:19:39.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:19:39 smithi090 ceph-mon[36989]: from='client.? 172.21.15.29:0/1404630693' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-18T01:19:39.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:19:39 smithi029 ceph-mon[29331]: from='client.14608 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T01:19:39.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:19:39 smithi029 ceph-mon[29331]: from='client.? 172.21.15.29:0/1404630693' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-18T01:19:40.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:19:40 smithi090 ceph-mon[36989]: pgmap v228: 65 pgs: 65 active+clean; 50 MiB data, 1.9 GiB used, 535 GiB / 536 GiB avail; 1.4 MiB/s rd, 4.3 MiB/s wr, 361 op/s 2024-03-18T01:19:40.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:19:40 smithi029 ceph-mon[29331]: pgmap v228: 65 pgs: 65 active+clean; 50 MiB data, 1.9 GiB used, 535 GiB / 536 GiB avail; 1.4 MiB/s rd, 4.3 MiB/s wr, 361 op/s 2024-03-18T01:19:41.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:19:41 smithi090 ceph-mon[36989]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-18T01:19:41.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:19:41 smithi029 ceph-mon[29331]: from='mgr.14208 172.21.15.29:0/2895636489' entity='mgr.smithi029.qviwhl' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-18T01:19:42.319 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:19:42 smithi029 ceph-mon[29331]: pgmap v229: 65 pgs: 65 active+clean; 63 MiB data, 1.9 GiB used, 534 GiB / 536 GiB avail; 2.4 MiB/s rd, 5.4 MiB/s wr, 399 op/s 2024-03-18T01:19:42.526 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Mar 18 01:19:42 smithi090 ceph-mon[36989]: pgmap v229: 65 pgs: 65 active+clean; 63 MiB data, 1.9 GiB used, 534 GiB / 536 GiB avail; 2.4 MiB/s rd, 5.4 MiB/s wr, 399 op/s 2024-03-18T01:19:43.181 INFO:tasks.workunit.client.1.smithi090.stderr:+ pushd ltp-full-20091231/testcases/kernel/fs/fsstress 2024-03-18T01:19:43.181 INFO:tasks.workunit.client.1.smithi090.stdout:~/cephtest/mnt.1/client.1/tmp/fsstress/ltp-full-20091231/testcases/kernel/fs/fsstress ~/cephtest/mnt.1/client.1/tmp/fsstress ~/cephtest/mnt.1/client.1/tmp 2024-03-18T01:19:43.182 INFO:tasks.workunit.client.1.smithi090.stderr:+ make 2024-03-18T01:19:43.296 INFO:tasks.workunit.client.1.smithi090.stdout:cc -DNO_XFS -I/home/ubuntu/cephtest/mnt.1/client.1/tmp/fsstress/ltp-full-20091231/testcases/kernel/fs/fsstress -D_LARGEFILE64_SOURCE -D_GNU_SOURCE -I../../../../include -I../../../../include -L../../../../lib fsstress.c -o fsstress 2024-03-18T01:20:05.143 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:20:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:20:04.826+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:20:05.143 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:20:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:20:04.826+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:20:05.143 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:20:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:20:04.826+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:20:05.143 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:20:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:20:04.842+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:20:05.143 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:20:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:20:04.842+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:20:05.143 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:20:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:20:04.842+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:20:06.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:20:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:20:05.873+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:20:06.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:20:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:20:05.873+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:20:06.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:20:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:20:05.873+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:20:06.319 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:20:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:20:05.822+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:20:06.319 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:20:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:20:05.822+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:20:06.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:20:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:20:05.822+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:20:07.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:20:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:20:06.904+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:20:07.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:20:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:20:06.904+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:20:07.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:20:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:20:06.904+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:20:07.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:20:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:20:06.788+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:20:07.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:20:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:20:06.788+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:20:07.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:20:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:20:06.788+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:20:07.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:20:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:20:06.659+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:20:07.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:20:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:20:06.659+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:20:07.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:20:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:20:06.659+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:20:08.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:20:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:20:07.896+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:20:08.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:20:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:20:07.896+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:20:08.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:20:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:20:07.896+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:20:08.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:20:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:20:07.753+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:20:08.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:20:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:20:07.753+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:20:08.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:20:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:20:07.753+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:20:08.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:20:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:20:07.651+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:20:08.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:20:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:20:07.651+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:20:08.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:20:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:20:07.651+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:20:08.881 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:20:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:20:08.708+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:20:08.881 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:20:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:20:08.708+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:20:08.881 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:20:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:20:08.708+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:20:08.881 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:20:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:20:08.623+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:20:08.881 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:20:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:20:08.623+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:20:08.881 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:20:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:20:08.623+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:20:09.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:20:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:20:08.880+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:20:09.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:20:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:20:08.880+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:20:09.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:20:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:20:08.880+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:20:10.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:20:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:20:09.701+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:20:10.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:20:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:20:09.701+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:20:10.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:20:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:20:09.701+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:20:10.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:20:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:20:09.662+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:20:10.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:20:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:20:09.662+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:20:10.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:20:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:20:09.662+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:20:10.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:20:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:20:09.852+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:20:10.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:20:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:20:09.852+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:20:10.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:20:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:20:09.852+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:20:11.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:20:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:20:10.696+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:20:11.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:20:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:20:10.696+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:20:11.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:20:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:20:10.696+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:20:11.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:20:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:20:10.742+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:20:11.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:20:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:20:10.742+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:20:11.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:20:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:20:10.742+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:20:11.489 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:20:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:20:10.822+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:20:11.489 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:20:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:20:10.822+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:20:11.489 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:20:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:20:10.822+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:20:11.740 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:20:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:20:11.739+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:20:11.740 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:20:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:20:11.739+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:20:11.740 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:20:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:20:11.739+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:20:12.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:20:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:20:11.813+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:20:12.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:20:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:20:11.813+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:20:12.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:20:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:20:11.813+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:20:12.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:20:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:20:11.763+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:20:12.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:20:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:20:11.763+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:20:12.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:20:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:20:11.763+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:20:13.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:20:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:20:12.770+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:20:13.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:20:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:20:12.770+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:20:13.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:20:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:20:12.770+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:20:13.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:20:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:20:12.815+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:20:13.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:20:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:20:12.815+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:20:13.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:20:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:20:12.815+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:20:13.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:20:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:20:12.775+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:20:13.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:20:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:20:12.775+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:20:13.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:20:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:20:12.775+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:20:13.800 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:20:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:20:13.727+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:20:13.800 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:20:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:20:13.727+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:20:13.800 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:20:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:20:13.727+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:20:14.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:20:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:20:13.847+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:20:14.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:20:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:20:13.847+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:20:14.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:20:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:20:13.847+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:20:14.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:20:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:20:13.800+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:20:14.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:20:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:20:13.800+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:20:14.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:20:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:20:13.800+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:20:15.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:20:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:20:14.879+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:20:15.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:20:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:20:14.879+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:20:15.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:20:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:20:14.879+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:20:15.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:20:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:20:14.695+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:20:15.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:20:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:20:14.695+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:20:15.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:20:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:20:14.695+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:20:15.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:20:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:20:14.840+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:20:15.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:20:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:20:14.840+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:20:15.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:20:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:20:14.840+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:20:16.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:20:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:20:15.827+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:20:16.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:20:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:20:15.827+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:20:16.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:20:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:20:15.827+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:20:16.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:20:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:20:15.869+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:20:16.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:20:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:20:15.869+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:20:16.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:20:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:20:15.869+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:20:16.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:20:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:20:15.658+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:20:16.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:20:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:20:15.658+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:20:16.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:20:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:20:15.658+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:20:17.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:20:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:20:16.834+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:20:17.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:20:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:20:16.834+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:20:17.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:20:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:20:16.834+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:20:17.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:20:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:20:16.825+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:20:17.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:20:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:20:16.825+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:20:17.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:20:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:20:16.825+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:20:17.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:20:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:20:16.645+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:20:17.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:20:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:20:16.645+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:20:17.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:20:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:20:16.645+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:20:18.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:20:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:20:17.814+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:20:18.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:20:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:20:17.814+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:20:18.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:20:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:20:17.814+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:20:18.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:20:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:20:17.782+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:20:18.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:20:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:20:17.782+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:20:18.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:20:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:20:17.782+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:20:18.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:20:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:20:17.681+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:20:18.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:20:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:20:17.681+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:20:18.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:20:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:20:17.681+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:20:19.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:20:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:20:18.768+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:20:19.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:20:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:20:18.768+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:20:19.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:20:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:20:18.768+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:20:19.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:20:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:20:18.830+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:20:19.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:20:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:20:18.830+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:20:19.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:20:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:20:18.830+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:20:19.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:20:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:20:18.666+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:20:19.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:20:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:20:18.666+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:20:19.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:20:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:20:18.666+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:20:20.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:20:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:20:19.807+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:20:20.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:20:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:20:19.807+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:20:20.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:20:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:20:19.807+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:20:20.071 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:20:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:20:19.873+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:20:20.071 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:20:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:20:19.873+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:20:20.071 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:20:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:20:19.873+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:20:20.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:20:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:20:19.703+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:20:20.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:20:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:20:19.703+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:20:20.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:20:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:20:19.703+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:20:21.068 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:20:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:20:20.900+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:20:21.068 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:20:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:20:20.900+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:20:21.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:20:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:20:20.900+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:20:21.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:20:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:20:20.746+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:20:21.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:20:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:20:20.746+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:20:21.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:20:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:20:20.746+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:20:21.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:20:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:20:20.827+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:20:21.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:20:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:20:20.827+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:20:21.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:20:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:20:20.827+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:20:22.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:20:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:20:21.777+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:20:22.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:20:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:20:21.777+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:20:22.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:20:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:20:21.777+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:20:22.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:20:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:20:21.830+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:20:22.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:20:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:20:21.830+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:20:22.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:20:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:20:21.830+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:20:22.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:20:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:20:21.942+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:20:22.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:20:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:20:21.942+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:20:22.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:20:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:20:21.942+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:20:23.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:20:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:20:22.934+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:20:23.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:20:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:20:22.934+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:20:23.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:20:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:20:22.934+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:20:23.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:20:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:20:22.770+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:20:23.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:20:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:20:22.770+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:20:23.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:20:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:20:22.770+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:20:23.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:20:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:20:22.787+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:20:23.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:20:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:20:22.787+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:20:23.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:20:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:20:22.787+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:20:24.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:20:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:20:23.759+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:20:24.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:20:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:20:23.759+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:20:24.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:20:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:20:23.759+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:20:24.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:20:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:20:23.771+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:20:24.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:20:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:20:23.771+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:20:24.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:20:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:20:23.771+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:20:24.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:20:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:20:23.946+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:20:24.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:20:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:20:23.946+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:20:24.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:20:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:20:23.946+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:20:25.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:20:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:20:24.742+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:20:25.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:20:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:20:24.742+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:20:25.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:20:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:20:24.742+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:20:25.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:20:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:20:24.975+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:20:25.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:20:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:20:24.975+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:20:25.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:20:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:20:24.975+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:20:25.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:20:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:20:24.767+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:20:25.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:20:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:20:24.767+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:20:25.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:20:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:20:24.767+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:20:26.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:20:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:20:25.704+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:20:26.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:20:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:20:25.704+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:20:26.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:20:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:20:25.704+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:20:26.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:20:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:20:25.946+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:20:26.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:20:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:20:25.946+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:20:26.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:20:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:20:25.946+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:20:26.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:20:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:20:25.755+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:20:26.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:20:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:20:25.755+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:20:26.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:20:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:20:25.755+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:20:26.955 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:20:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:20:26.715+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:20:26.955 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:20:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:20:26.715+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:20:26.955 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:20:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:20:26.715+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:20:26.956 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:20:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:20:26.701+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:20:26.956 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:20:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:20:26.701+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:20:26.956 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:20:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:20:26.701+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:20:27.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:20:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:20:26.955+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:20:27.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:20:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:20:26.955+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:20:27.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:20:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:20:26.955+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:20:28.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:20:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:20:27.727+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:20:28.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:20:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:20:27.727+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:20:28.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:20:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:20:27.727+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:20:28.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:20:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:20:27.719+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:20:28.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:20:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:20:27.719+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:20:28.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:20:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:20:27.719+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:20:28.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:20:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:20:27.921+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:20:28.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:20:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:20:27.921+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:20:28.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:20:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:20:27.921+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:20:29.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:20:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:20:28.911+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:20:29.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:20:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:20:28.911+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:20:29.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:20:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:20:28.911+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:20:29.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:20:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:20:28.698+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:20:29.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:20:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:20:28.698+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:20:29.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:20:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:20:28.698+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:20:29.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:20:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:20:28.768+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:20:29.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:20:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:20:28.768+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:20:29.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:20:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:20:28.768+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:20:30.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:20:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:20:29.887+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:20:30.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:20:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:20:29.887+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:20:30.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:20:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:20:29.887+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:20:30.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:20:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:20:29.758+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:20:30.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:20:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:20:29.758+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:20:30.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:20:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:20:29.758+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:20:30.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:20:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:20:29.727+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:20:30.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:20:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:20:29.727+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:20:30.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:20:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:20:29.727+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:20:31.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:20:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:20:30.892+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:20:31.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:20:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:20:30.892+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:20:31.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:20:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:20:30.892+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:20:31.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:20:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:20:30.741+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:20:31.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:20:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:20:30.741+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:20:31.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:20:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:20:30.741+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:20:31.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:20:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:20:30.682+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:20:31.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:20:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:20:30.682+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:20:31.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:20:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:20:30.682+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:20:31.911 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:20:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:20:31.758+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:20:31.911 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:20:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:20:31.758+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:20:31.911 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:20:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:20:31.758+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:20:31.911 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:20:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:20:31.649+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:20:31.911 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:20:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:20:31.649+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:20:31.911 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:20:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:20:31.649+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:20:32.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:20:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:20:31.910+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:20:32.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:20:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:20:31.910+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:20:32.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:20:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:20:31.910+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:20:32.906 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:20:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:20:32.771+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:20:32.907 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:20:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:20:32.771+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:20:32.907 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:20:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:20:32.771+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:20:32.907 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:20:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:20:32.601+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:20:32.907 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:20:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:20:32.601+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:20:32.907 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:20:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:20:32.601+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:20:33.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:20:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:20:32.906+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:20:33.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:20:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:20:32.906+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:20:33.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:20:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:20:32.906+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:20:33.762 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:20:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:20:33.610+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:20:33.763 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:20:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:20:33.610+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:20:33.763 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:20:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:20:33.610+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:20:34.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:20:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:20:33.934+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:20:34.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:20:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:20:33.934+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:20:34.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:20:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:20:33.934+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:20:34.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:20:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:20:33.762+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:20:34.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:20:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:20:33.762+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:20:34.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:20:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:20:33.762+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:20:34.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:20:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:20:34.738+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:20:34.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:20:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:20:34.738+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:20:34.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:20:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:20:34.738+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:20:34.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:20:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:20:34.565+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:20:34.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:20:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:20:34.565+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:20:34.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:20:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:20:34.565+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:20:35.256 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:20:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:20:34.963+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:20:35.256 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:20:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:20:34.963+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:20:35.256 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:20:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:20:34.963+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:20:35.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:20:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:20:35.540+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:20:35.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:20:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:20:35.540+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:20:35.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:20:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:20:35.540+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:20:35.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:20:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:20:35.691+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:20:35.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:20:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:20:35.691+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:20:35.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:20:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:20:35.691+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:20:36.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:20:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:20:35.992+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:20:36.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:20:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:20:35.992+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:20:36.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:20:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:20:35.992+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:20:36.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:20:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:20:36.505+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:20:36.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:20:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:20:36.505+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:20:36.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:20:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:20:36.505+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:20:36.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:20:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:20:36.726+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:20:36.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:20:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:20:36.726+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:20:36.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:20:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:20:36.726+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:20:37.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:20:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:20:36.969+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:20:37.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:20:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:20:36.969+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:20:37.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:20:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:20:36.969+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:20:37.770 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:20:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:20:37.498+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:20:37.770 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:20:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:20:37.498+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:20:37.770 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:20:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:20:37.498+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:20:38.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:20:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:20:37.769+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:20:38.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:20:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:20:37.769+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:20:38.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:20:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:20:37.769+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:20:38.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:20:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:20:37.995+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:20:38.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:20:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:20:37.995+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:20:38.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:20:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:20:37.995+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:20:38.760 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:20:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:20:38.459+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:20:38.760 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:20:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:20:38.459+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:20:38.760 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:20:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:20:38.459+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:20:39.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:20:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:20:38.976+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:20:39.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:20:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:20:38.976+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:20:39.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:20:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:20:38.976+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:20:39.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:20:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:20:38.759+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:20:39.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:20:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:20:38.759+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:20:39.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:20:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:20:38.759+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:20:39.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:20:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:20:39.733+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:20:39.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:20:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:20:39.733+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:20:39.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:20:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:20:39.733+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:20:39.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:20:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:20:39.498+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:20:39.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:20:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:20:39.498+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:20:39.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:20:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:20:39.499+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:20:40.257 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:20:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:20:39.932+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:20:40.258 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:20:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:20:39.932+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:20:40.258 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:20:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:20:39.932+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:20:40.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:20:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:20:40.694+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:20:40.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:20:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:20:40.694+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:20:40.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:20:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:20:40.694+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:20:40.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:20:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:20:40.538+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:20:40.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:20:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:20:40.539+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:20:40.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:20:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:20:40.539+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:20:41.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:20:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:20:40.956+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:20:41.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:20:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:20:40.956+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:20:41.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:20:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:20:40.956+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:20:41.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:20:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:20:41.533+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:20:41.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:20:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:20:41.533+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:20:41.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:20:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:20:41.533+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:20:41.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:20:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:20:41.711+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:20:41.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:20:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:20:41.711+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:20:41.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:20:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:20:41.711+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:20:42.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:20:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:20:41.989+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:20:42.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:20:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:20:41.989+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:20:42.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:20:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:20:41.989+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:20:42.969 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:20:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:20:42.571+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:20:42.970 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:20:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:20:42.571+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:20:42.970 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:20:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:20:42.571+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:20:42.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:20:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:20:42.698+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:20:42.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:20:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:20:42.698+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:20:42.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:20:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:20:42.698+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:20:43.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:20:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:20:42.969+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:20:43.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:20:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:20:42.969+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:20:43.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:20:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:20:42.969+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:20:43.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:20:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:20:43.665+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:20:43.887 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:20:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:20:43.665+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:20:43.888 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:20:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:20:43.665+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:20:43.888 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:20:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:20:43.621+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:20:43.888 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:20:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:20:43.621+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:20:43.888 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:20:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:20:43.621+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:20:44.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:20:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:20:44.001+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:20:44.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:20:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:20:44.001+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:20:44.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:20:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:20:44.001+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:20:45.042 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:20:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:20:44.664+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:20:45.043 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:20:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:20:44.664+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:20:45.043 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:20:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:20:44.664+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:20:45.043 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:20:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:20:44.574+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:20:45.043 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:20:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:20:44.574+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:20:45.043 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:20:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:20:44.574+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:20:45.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:20:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:20:45.042+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:20:45.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:20:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:20:45.042+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:20:45.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:20:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:20:45.042+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:20:45.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:20:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:20:45.641+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:20:45.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:20:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:20:45.641+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:20:45.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:20:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:20:45.641+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:20:45.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:20:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:20:45.526+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:20:45.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:20:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:20:45.526+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:20:45.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:20:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:20:45.526+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:20:46.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:20:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:20:45.998+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:20:46.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:20:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:20:45.998+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:20:46.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:20:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:20:45.998+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:20:46.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:20:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:20:46.521+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:20:46.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:20:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:20:46.521+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:20:46.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:20:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:20:46.521+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:20:46.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:20:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:20:46.668+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:20:46.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:20:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:20:46.668+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:20:46.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:20:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:20:46.668+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:20:47.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:20:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:20:46.984+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:20:47.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:20:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:20:46.984+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:20:47.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:20:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:20:46.984+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:20:47.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:20:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:20:47.706+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:20:47.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:20:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:20:47.706+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:20:47.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:20:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:20:47.706+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:20:47.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:20:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:20:47.541+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:20:47.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:20:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:20:47.541+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:20:47.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:20:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:20:47.541+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:20:48.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:20:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:20:47.948+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:20:48.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:20:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:20:47.948+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:20:48.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:20:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:20:47.948+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:20:48.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:20:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:20:48.532+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:20:48.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:20:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:20:48.532+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:20:48.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:20:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:20:48.532+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:20:48.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:20:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:20:48.675+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:20:48.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:20:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:20:48.675+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:20:48.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:20:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:20:48.675+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:20:49.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:20:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:20:48.974+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:20:49.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:20:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:20:48.974+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:20:49.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:20:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:20:48.974+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:20:49.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:20:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:20:49.489+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:20:49.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:20:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:20:49.489+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:20:49.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:20:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:20:49.489+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:20:49.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:20:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:20:49.715+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:20:49.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:20:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:20:49.715+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:20:49.821 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:20:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:20:49.715+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:20:50.260 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:20:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:20:49.972+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:20:50.260 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:20:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:20:49.972+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:20:50.261 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:20:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:20:49.972+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:20:50.569 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:20:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:20:50.494+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:20:50.569 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:20:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:20:50.494+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:20:50.569 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:20:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:20:50.494+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:20:51.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:20:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:20:50.931+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:20:51.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:20:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:20:50.931+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:20:51.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:20:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:20:50.931+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:20:51.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:20:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:20:50.683+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:20:51.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:20:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:20:50.683+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:20:51.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:20:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:20:50.683+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:20:51.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:20:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:20:51.455+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:20:51.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:20:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:20:51.455+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:20:51.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:20:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:20:51.455+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:20:51.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:20:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:20:51.648+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:20:51.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:20:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:20:51.648+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:20:51.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:20:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:20:51.648+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:20:52.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:20:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:20:51.883+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:20:52.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:20:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:20:51.883+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:20:52.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:20:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:20:51.883+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:20:52.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:20:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:20:52.493+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:20:52.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:20:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:20:52.493+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:20:52.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:20:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:20:52.493+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:20:52.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:20:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:20:52.643+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:20:52.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:20:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:20:52.643+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:20:52.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:20:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:20:52.643+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:20:53.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:20:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:20:52.846+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:20:53.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:20:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:20:52.846+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:20:53.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:20:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:20:52.846+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:20:53.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:20:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:20:53.647+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:20:53.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:20:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:20:53.647+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:20:53.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:20:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:20:53.647+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:20:53.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:20:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:20:53.533+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:20:53.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:20:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:20:53.533+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:20:53.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:20:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:20:53.533+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:20:54.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:20:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:20:53.850+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:20:54.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:20:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:20:53.850+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:20:54.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:20:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:20:53.850+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:20:54.807 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:20:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:20:54.619+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:20:54.807 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:20:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:20:54.619+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:20:54.807 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:20:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:20:54.619+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:20:54.807 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:20:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:20:54.533+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:20:54.807 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:20:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:20:54.533+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:20:54.807 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:20:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:20:54.533+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:20:55.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:20:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:20:54.805+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:20:55.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:20:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:20:54.805+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:20:55.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:20:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:20:54.805+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:20:55.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:20:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:20:55.612+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:20:55.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:20:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:20:55.612+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:20:55.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:20:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:20:55.612+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:20:55.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:20:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:20:55.538+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:20:55.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:20:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:20:55.538+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:20:55.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:20:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:20:55.538+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:20:56.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:20:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:20:55.817+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:20:56.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:20:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:20:55.818+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:20:56.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:20:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:20:55.818+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:20:56.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:20:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:20:56.641+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:20:56.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:20:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:20:56.641+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:20:56.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:20:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:20:56.641+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:20:56.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:20:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:20:56.548+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:20:56.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:20:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:20:56.548+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:20:56.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:20:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:20:56.548+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:20:57.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:20:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:20:56.840+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:20:57.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:20:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:20:56.840+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:20:57.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:20:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:20:56.840+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:20:57.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:20:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:20:57.690+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:20:57.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:20:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:20:57.690+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:20:57.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:20:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:20:57.690+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:20:57.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:20:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:20:57.540+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:20:57.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:20:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:20:57.540+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:20:57.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:20:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:20:57.540+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:20:58.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:20:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:20:57.848+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:20:58.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:20:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:20:57.848+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:20:58.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:20:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:20:57.848+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:20:58.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:20:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:20:58.532+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:20:58.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:20:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:20:58.532+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:20:58.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:20:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:20:58.532+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:20:58.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:20:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:20:58.717+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:20:58.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:20:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:20:58.717+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:20:58.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:20:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:20:58.717+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:20:59.115 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:20:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:20:58.857+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:20:59.115 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:20:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:20:58.857+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:20:59.115 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:20:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:20:58.857+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:20:59.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:20:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:20:59.761+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:20:59.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:20:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:20:59.761+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:20:59.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:20:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:20:59.761+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:20:59.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:20:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:20:59.518+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:20:59.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:20:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:20:59.518+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:20:59.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:20:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:20:59.518+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:21:00.146 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:20:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:20:59.851+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:21:00.147 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:20:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:20:59.851+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:21:00.147 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:20:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:20:59.851+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:21:00.808 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:21:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:21:00.476+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:21:00.808 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:21:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:21:00.476+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:21:00.809 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:21:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:21:00.476+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:21:01.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:21:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:21:00.807+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:21:01.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:21:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:21:00.807+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:21:01.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:21:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:21:00.807+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:21:01.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:21:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:21:00.810+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:21:01.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:21:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:21:00.810+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:21:01.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:21:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:21:00.810+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:21:01.785 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:21:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:21:01.445+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:21:01.785 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:21:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:21:01.445+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:21:01.785 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:21:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:21:01.445+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:21:02.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:21:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:21:01.796+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:21:02.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:21:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:21:01.796+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:21:02.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:21:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:21:01.796+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:21:02.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:21:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:21:01.783+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:21:02.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:21:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:21:01.783+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:21:02.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:21:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:21:01.783+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:21:02.747 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:21:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:21:02.396+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:21:02.747 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:21:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:21:02.396+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:21:02.747 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:21:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:21:02.396+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:21:03.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:21:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:21:02.757+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:21:03.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:21:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:21:02.757+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:21:03.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:21:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:21:02.757+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:21:03.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:21:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:21:02.783+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:21:03.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:21:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:21:02.783+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:21:03.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:21:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:21:02.783+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:21:03.761 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:21:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:21:03.403+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:21:03.762 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:21:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:21:03.403+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:21:03.762 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:21:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:21:03.403+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:21:04.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:21:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:21:03.760+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:21:04.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:21:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:21:03.760+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:21:04.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:21:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:21:03.760+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:21:04.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:21:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:21:03.804+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:21:04.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:21:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:21:03.804+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:21:04.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:21:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:21:03.804+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:21:04.752 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:21:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:21:04.372+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:21:04.752 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:21:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:21:04.372+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:21:04.753 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:21:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:21:04.372+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:21:05.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:21:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:21:04.824+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:21:05.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:21:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:21:04.824+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:21:05.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:21:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:21:04.824+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:21:05.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:21:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:21:04.752+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:21:05.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:21:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:21:04.752+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:21:05.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:21:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:21:04.752+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:21:05.569 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:21:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:21:05.395+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:21:05.569 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:21:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:21:05.395+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:21:05.569 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:21:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:21:05.395+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:21:06.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:21:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:21:05.730+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:21:06.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:21:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:21:05.730+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:21:06.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:21:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:21:05.730+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:21:06.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:21:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:21:05.816+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:21:06.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:21:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:21:05.816+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:21:06.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:21:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:21:05.816+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:21:06.710 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:21:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:21:06.436+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:21:06.711 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:21:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:21:06.436+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:21:06.711 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:21:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:21:06.436+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:21:07.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:21:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:21:06.710+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:21:07.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:21:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:21:06.710+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:21:07.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:21:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:21:06.710+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:21:07.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:21:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:21:06.768+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:21:07.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:21:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:21:06.768+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:21:07.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:21:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:21:06.768+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:21:07.754 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:21:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:21:07.395+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:21:07.754 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:21:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:21:07.395+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:21:07.754 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:21:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:21:07.395+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:21:08.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:21:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:21:07.753+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:21:08.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:21:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:21:07.753+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:21:08.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:21:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:21:07.753+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:21:08.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:21:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:21:07.778+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:21:08.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:21:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:21:07.778+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:21:08.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:21:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:21:07.778+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:21:08.748 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:21:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:21:08.389+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:21:08.749 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:21:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:21:08.389+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:21:08.749 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:21:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:21:08.389+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:21:09.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:21:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:21:08.748+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:21:09.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:21:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:21:08.748+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:21:09.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:21:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:21:08.748+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:21:09.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:21:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:21:08.764+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:21:09.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:21:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:21:08.764+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:21:09.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:21:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:21:08.764+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:21:09.723 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:21:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:21:09.392+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:21:09.723 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:21:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:21:09.392+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:21:09.723 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:21:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:21:09.392+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:21:10.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:21:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:21:09.723+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:21:10.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:21:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:21:09.723+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:21:10.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:21:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:21:09.723+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:21:10.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:21:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:21:09.803+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:21:10.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:21:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:21:09.803+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:21:10.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:21:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:21:09.803+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:21:10.684 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:21:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:21:10.432+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:21:10.684 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:21:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:21:10.432+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:21:10.684 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:21:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:21:10.432+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:21:11.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:21:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:21:10.683+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:21:11.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:21:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:21:10.683+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:21:11.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:21:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:21:10.683+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:21:11.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:21:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:21:10.782+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:21:11.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:21:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:21:10.782+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:21:11.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:21:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:21:10.782+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:21:11.810 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:21:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:21:11.646+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:21:11.810 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:21:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:21:11.646+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:21:11.810 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:21:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:21:11.646+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:21:11.810 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:21:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:21:11.465+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:21:11.810 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:21:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:21:11.466+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:21:11.810 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:21:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:21:11.466+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:21:12.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:21:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:21:11.809+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:21:12.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:21:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:21:11.809+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:21:12.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:21:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:21:11.809+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:21:12.774 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:21:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:21:12.457+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:21:12.774 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:21:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:21:12.457+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:21:12.774 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:21:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:21:12.457+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:21:12.774 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:21:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:21:12.664+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:21:12.775 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:21:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:21:12.664+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:21:12.775 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:21:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:21:12.664+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:21:13.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:21:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:21:12.773+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:21:13.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:21:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:21:12.773+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:21:13.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:21:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:21:12.773+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:21:13.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:21:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:21:13.681+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:21:13.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:21:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:21:13.681+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:21:13.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:21:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:21:13.681+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:21:13.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:21:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:21:13.488+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:21:13.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:21:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:21:13.489+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:21:13.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:21:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:21:13.489+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:21:13.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:21:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:21:13.738+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:21:13.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:21:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:21:13.738+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:21:13.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:21:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:21:13.738+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:21:14.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:21:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:21:14.661+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:21:14.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:21:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:21:14.661+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:21:14.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:21:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:21:14.661+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:21:14.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:21:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:21:14.695+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:21:14.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:21:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:21:14.695+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:21:14.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:21:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:21:14.695+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:21:14.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:21:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:21:14.513+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:21:14.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:21:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:21:14.513+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:21:14.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:21:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:21:14.513+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:21:15.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:21:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:21:15.655+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:21:15.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:21:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:21:15.655+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:21:15.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:21:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:21:15.655+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:21:15.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:21:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:21:15.677+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:21:15.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:21:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:21:15.677+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:21:15.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:21:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:21:15.677+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:21:15.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:21:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:21:15.515+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:21:15.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:21:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:21:15.515+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:21:15.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:21:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:21:15.515+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:21:16.822 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:21:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:21:16.711+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:21:16.822 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:21:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:21:16.711+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:21:16.822 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:21:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:21:16.711+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:21:16.823 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:21:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:21:16.623+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:21:16.823 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:21:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:21:16.623+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:21:16.823 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:21:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:21:16.623+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:21:16.823 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:21:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:21:16.525+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:21:16.823 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:21:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:21:16.525+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:21:16.823 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:21:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:21:16.525+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:21:17.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:21:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:21:17.757+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:21:17.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:21:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:21:17.757+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:21:17.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:21:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:21:17.757+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:21:17.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:21:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:21:17.605+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:21:17.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:21:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:21:17.605+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:21:17.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:21:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:21:17.605+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:21:17.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:21:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:21:17.531+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:21:17.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:21:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:21:17.531+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:21:17.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:21:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:21:17.531+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:21:19.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:21:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:21:18.581+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:21:19.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:21:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:21:18.581+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:21:19.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:21:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:21:18.581+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:21:19.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:21:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:21:18.724+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:21:19.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:21:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:21:18.724+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:21:19.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:21:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:21:18.724+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:21:19.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:21:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:21:18.580+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:21:19.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:21:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:21:18.580+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:21:19.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:21:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:21:18.580+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:21:19.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:21:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:21:19.601+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:21:19.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:21:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:21:19.601+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:21:19.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:21:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:21:19.601+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:21:19.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:21:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:21:19.543+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:21:19.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:21:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:21:19.543+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:21:19.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:21:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:21:19.543+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:21:19.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:21:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:21:19.714+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:21:19.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:21:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:21:19.714+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:21:19.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:21:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:21:19.714+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:21:21.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:21:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:21:20.573+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:21:21.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:21:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:21:20.573+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:21:21.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:21:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:21:20.573+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:21:21.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:21:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:21:20.583+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:21:21.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:21:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:21:20.583+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:21:21.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:21:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:21:20.583+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:21:21.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:21:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:21:20.715+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:21:21.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:21:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:21:20.715+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:21:21.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:21:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:21:20.715+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:21:21.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:21:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:21:21.570+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:21:21.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:21:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:21:21.570+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:21:21.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:21:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:21:21.570+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:21:21.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:21:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:21:21.545+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:21:21.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:21:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:21:21.545+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:21:21.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:21:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:21:21.545+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:21:21.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:21:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:21:21.736+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:21:21.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:21:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:21:21.736+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:21:21.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:21:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:21:21.736+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:21:22.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:21:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:21:22.591+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:21:22.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:21:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:21:22.591+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:21:22.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:21:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:21:22.591+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:21:22.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:21:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:21:22.525+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:21:22.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:21:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:21:22.525+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:21:22.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:21:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:21:22.525+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:21:22.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:21:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:21:22.765+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:21:22.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:21:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:21:22.765+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:21:22.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:21:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:21:22.765+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:21:23.789 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:21:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:21:23.507+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:21:23.790 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:21:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:21:23.507+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:21:23.790 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:21:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:21:23.507+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:21:23.790 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:21:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:21:23.606+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:21:23.790 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:21:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:21:23.606+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:21:23.790 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:21:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:21:23.606+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:21:24.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:21:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:21:23.789+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:21:24.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:21:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:21:23.789+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:21:24.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:21:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:21:23.789+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:21:24.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:21:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:21:24.646+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:21:24.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:21:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:21:24.646+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:21:24.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:21:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:21:24.646+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:21:24.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:21:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:21:24.527+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:21:24.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:21:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:21:24.527+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:21:24.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:21:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:21:24.527+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:21:25.150 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:21:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:21:24.828+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:21:25.150 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:21:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:21:24.828+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:21:25.150 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:21:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:21:24.828+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:21:25.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:21:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:21:25.606+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:21:25.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:21:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:21:25.606+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:21:25.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:21:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:21:25.606+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:21:25.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:21:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:21:25.541+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:21:25.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:21:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:21:25.541+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:21:25.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:21:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:21:25.541+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:21:26.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:21:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:21:25.823+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:21:26.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:21:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:21:25.823+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:21:26.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:21:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:21:25.823+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:21:26.790 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:21:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:21:26.540+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:21:26.790 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:21:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:21:26.540+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:21:26.790 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:21:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:21:26.540+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:21:26.791 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:21:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:21:26.656+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:21:26.791 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:21:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:21:26.656+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:21:26.791 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:21:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:21:26.656+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:21:27.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:21:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:21:26.790+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:21:27.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:21:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:21:26.790+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:21:27.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:21:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:21:26.790+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:21:28.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:21:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:21:27.811+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:21:28.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:21:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:21:27.811+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:21:28.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:21:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:21:27.811+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:21:28.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:21:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:21:27.654+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:21:28.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:21:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:21:27.654+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:21:28.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:21:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:21:27.654+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:21:28.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:21:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:21:27.588+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:21:28.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:21:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:21:27.588+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:21:28.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:21:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:21:27.588+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:21:29.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:21:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:21:28.699+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:21:29.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:21:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:21:28.699+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:21:29.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:21:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:21:28.699+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:21:29.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:21:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:21:28.780+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:21:29.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:21:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:21:28.780+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:21:29.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:21:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:21:28.780+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:21:29.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:21:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:21:28.609+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:21:29.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:21:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:21:28.609+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:21:29.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:21:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:21:28.609+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:21:30.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:21:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:21:29.696+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:21:30.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:21:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:21:29.696+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:21:30.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:21:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:21:29.696+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:21:30.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:21:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:21:29.639+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:21:30.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:21:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:21:29.639+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:21:30.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:21:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:21:29.639+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:21:30.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:21:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:21:29.826+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:21:30.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:21:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:21:29.826+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:21:30.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:21:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:21:29.826+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:21:31.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:21:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:21:30.667+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:21:31.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:21:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:21:30.667+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:21:31.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:21:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:21:30.667+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:21:31.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:21:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:21:30.784+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:21:31.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:21:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:21:30.784+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:21:31.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:21:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:21:30.784+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:21:31.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:21:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:21:30.727+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:21:31.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:21:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:21:30.727+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:21:31.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:21:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:21:30.727+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:21:32.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:21:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:21:31.772+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:21:32.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:21:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:21:31.772+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:21:32.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:21:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:21:31.772+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:21:32.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:21:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:21:31.643+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:21:32.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:21:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:21:31.643+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:21:32.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:21:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:21:31.643+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:21:32.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:21:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:21:31.752+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:21:32.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:21:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:21:31.752+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:21:32.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:21:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:21:31.752+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:21:33.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:21:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:21:32.747+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:21:33.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:21:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:21:32.747+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:21:33.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:21:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:21:32.747+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:21:33.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:21:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:21:32.733+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:21:33.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:21:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:21:32.733+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:21:33.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:21:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:21:32.733+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:21:33.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:21:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:21:32.651+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:21:33.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:21:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:21:32.651+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:21:33.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:21:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:21:32.651+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:21:33.778 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:21:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:21:33.693+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:21:33.778 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:21:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:21:33.693+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:21:33.778 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:21:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:21:33.693+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:21:34.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:21:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:21:33.789+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:21:34.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:21:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:21:33.789+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:21:34.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:21:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:21:33.789+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:21:34.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:21:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:21:33.777+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:21:34.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:21:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:21:33.777+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:21:34.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:21:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:21:33.777+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:21:35.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:21:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:21:34.784+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:21:35.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:21:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:21:34.784+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:21:35.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:21:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:21:34.784+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:21:35.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:21:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:21:34.751+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:21:35.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:21:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:21:34.751+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:21:35.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:21:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:21:34.751+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:21:35.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:21:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:21:34.735+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:21:35.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:21:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:21:34.735+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:21:35.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:21:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:21:34.735+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:21:36.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:21:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:21:35.712+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:21:36.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:21:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:21:35.712+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:21:36.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:21:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:21:35.712+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:21:36.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:21:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:21:35.797+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:21:36.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:21:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:21:35.797+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:21:36.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:21:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:21:35.797+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:21:36.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:21:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:21:35.768+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:21:36.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:21:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:21:35.768+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:21:36.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:21:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:21:35.768+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:21:36.795 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:21:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:21:36.754+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:21:36.795 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:21:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:21:36.754+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:21:36.795 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:21:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:21:36.754+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:21:37.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:21:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:21:36.812+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:21:37.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:21:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:21:36.812+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:21:37.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:21:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:21:36.812+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:21:37.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:21:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:21:36.794+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:21:37.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:21:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:21:36.794+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:21:37.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:21:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:21:36.794+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:21:38.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:21:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:21:37.741+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:21:38.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:21:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:21:37.741+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:21:38.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:21:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:21:37.741+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:21:38.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:21:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:21:37.818+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:21:38.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:21:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:21:37.818+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:21:38.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:21:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:21:37.818+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:21:38.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:21:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:21:37.748+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:21:38.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:21:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:21:37.749+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:21:38.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:21:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:21:37.749+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:21:39.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:21:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:21:38.726+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:21:39.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:21:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:21:38.726+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:21:39.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:21:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:21:38.726+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:21:39.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:21:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:21:38.865+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:21:39.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:21:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:21:38.865+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:21:39.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:21:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:21:38.865+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:21:39.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:21:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:21:38.730+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:21:39.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:21:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:21:38.730+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:21:39.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:21:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:21:38.730+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:21:40.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:21:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:21:39.751+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:21:40.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:21:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:21:39.751+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:21:40.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:21:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:21:39.751+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:21:40.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:21:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:21:39.891+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:21:40.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:21:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:21:39.891+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:21:40.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:21:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:21:39.891+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:21:40.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:21:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:21:39.723+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:21:40.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:21:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:21:39.723+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:21:40.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:21:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:21:39.723+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:21:41.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:21:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:21:40.682+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:21:41.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:21:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:21:40.682+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:21:41.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:21:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:21:40.682+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:21:41.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:21:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:21:40.867+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:21:41.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:21:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:21:40.867+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:21:41.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:21:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:21:40.867+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:21:41.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:21:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:21:40.741+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:21:41.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:21:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:21:40.741+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:21:41.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:21:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:21:40.741+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:21:42.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:21:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:21:41.719+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:21:42.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:21:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:21:41.719+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:21:42.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:21:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:21:41.719+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:21:42.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:21:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:21:41.824+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:21:42.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:21:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:21:41.824+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:21:42.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:21:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:21:41.824+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:21:42.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:21:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:21:41.701+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:21:42.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:21:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:21:41.701+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:21:42.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:21:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:21:41.701+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:21:43.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:21:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:21:42.758+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:21:43.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:21:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:21:42.758+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:21:43.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:21:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:21:42.758+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:21:43.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:21:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:21:42.661+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:21:43.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:21:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:21:42.661+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:21:43.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:21:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:21:42.661+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:21:43.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:21:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:21:42.814+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:21:43.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:21:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:21:42.814+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:21:43.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:21:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:21:42.814+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:21:43.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:21:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:21:43.709+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:21:43.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:21:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:21:43.709+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:21:43.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:21:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:21:43.709+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:21:43.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:21:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:21:43.681+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:21:43.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:21:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:21:43.681+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:21:43.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:21:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:21:43.681+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:21:44.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:21:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:21:43.855+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:21:44.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:21:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:21:43.855+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:21:44.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:21:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:21:43.855+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:21:45.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:21:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:21:44.868+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:21:45.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:21:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:21:44.868+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:21:45.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:21:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:21:44.868+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:21:45.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:21:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:21:44.664+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:21:45.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:21:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:21:44.664+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:21:45.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:21:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:21:44.664+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:21:45.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:21:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:21:44.645+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:21:45.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:21:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:21:44.645+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:21:45.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:21:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:21:44.645+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:21:46.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:21:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:21:45.879+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:21:46.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:21:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:21:45.879+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:21:46.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:21:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:21:45.879+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:21:46.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:21:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:21:45.684+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:21:46.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:21:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:21:45.684+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:21:46.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:21:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:21:45.684+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:21:46.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:21:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:21:45.684+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:21:46.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:21:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:21:45.684+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:21:46.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:21:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:21:45.684+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:21:47.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:21:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:21:46.903+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:21:47.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:21:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:21:46.903+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:21:47.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:21:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:21:46.903+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:21:47.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:21:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:21:46.715+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:21:47.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:21:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:21:46.715+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:21:47.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:21:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:21:46.715+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:21:47.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:21:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:21:46.719+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:21:47.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:21:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:21:46.719+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:21:47.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:21:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:21:46.719+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:21:47.951 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:21:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:21:47.683+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:21:47.951 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:21:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:21:47.683+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:21:47.951 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:21:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:21:47.683+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:21:47.951 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:21:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:21:47.699+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:21:47.952 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:21:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:21:47.699+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:21:47.952 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:21:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:21:47.699+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:21:48.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:21:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:21:47.950+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:21:48.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:21:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:21:47.950+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:21:48.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:21:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:21:47.950+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:21:49.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:21:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:21:48.900+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:21:49.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:21:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:21:48.900+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:21:49.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:21:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:21:48.900+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:21:49.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:21:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:21:48.711+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:21:49.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:21:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:21:48.711+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:21:49.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:21:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:21:48.711+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:21:49.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:21:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:21:48.721+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:21:49.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:21:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:21:48.721+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:21:49.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:21:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:21:48.721+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:21:50.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:21:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:21:49.895+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:21:50.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:21:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:21:49.895+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:21:50.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:21:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:21:49.895+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:21:50.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:21:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:21:49.737+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:21:50.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:21:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:21:49.737+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:21:50.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:21:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:21:49.737+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:21:50.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:21:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:21:49.673+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:21:50.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:21:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:21:49.673+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:21:50.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:21:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:21:49.673+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:21:50.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:21:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:21:50.142+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 8 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:21:50.931 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:21:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:21:50.650+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:21:50.931 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:21:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:21:50.650+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:21:50.931 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:21:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:21:50.650+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:21:50.931 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:21:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:21:50.764+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:21:50.931 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:21:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:21:50.764+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:21:50.931 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:21:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:21:50.764+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:21:51.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:21:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:21:50.930+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:21:51.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:21:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:21:50.930+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:21:51.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:21:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:21:50.930+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:21:51.885 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:21:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:21:51.677+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:21:51.885 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:21:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:21:51.677+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:21:51.885 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:21:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:21:51.677+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:21:51.886 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:21:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:21:51.765+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:21:51.886 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:21:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:21:51.765+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:21:51.886 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:21:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:21:51.765+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:21:52.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:21:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:21:51.884+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:21:52.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:21:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:21:51.884+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:21:52.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:21:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:21:51.884+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:21:53.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:21:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:21:52.868+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:21:53.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:21:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:21:52.868+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:21:53.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:21:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:21:52.868+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:21:53.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:21:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:21:52.715+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:21:53.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:21:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:21:52.715+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:21:53.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:21:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:21:52.715+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:21:53.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:21:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:21:52.700+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:21:53.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:21:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:21:52.700+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:21:53.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:21:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:21:52.700+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:21:53.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:21:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:21:53.680+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:21:53.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:21:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:21:53.680+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:21:53.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:21:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:21:53.680+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:21:53.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:21:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:21:53.747+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:21:53.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:21:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:21:53.747+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:21:53.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:21:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:21:53.747+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:21:54.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:21:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:21:53.898+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:21:54.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:21:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:21:53.898+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:21:54.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:21:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:21:53.898+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:21:54.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:21:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:21:54.708+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:21:54.971 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:21:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:21:54.708+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:21:54.971 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:21:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:21:54.708+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:21:54.971 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:21:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:21:54.917+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:21:54.971 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:21:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:21:54.917+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:21:54.971 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:21:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:21:54.917+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:21:54.971 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:21:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:21:54.692+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:21:54.971 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:21:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:21:54.692+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:21:54.971 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:21:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:21:54.692+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:21:55.265 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:21:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:21:55.142+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 17 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:21:56.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:21:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:21:55.877+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:21:56.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:21:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:21:55.877+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:21:56.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:21:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:21:55.877+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:21:56.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:21:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:21:55.723+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:21:56.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:21:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:21:55.723+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:21:56.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:21:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:21:55.723+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:21:56.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:21:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:21:55.677+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:21:56.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:21:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:21:55.677+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:21:56.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:21:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:21:55.677+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:21:57.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:21:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:21:56.773+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:21:57.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:21:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:21:56.773+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:21:57.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:21:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:21:56.773+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:21:57.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:21:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:21:56.703+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:21:57.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:21:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:21:56.703+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:21:57.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:21:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:21:56.703+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:21:57.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:21:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:21:56.862+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:21:57.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:21:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:21:56.862+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:21:57.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:21:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:21:56.862+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:21:58.043 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:21:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:21:57.688+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:21:58.043 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:21:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:21:57.688+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:21:58.043 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:21:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:21:57.688+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:21:58.044 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:21:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:21:57.747+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:21:58.044 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:21:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:21:57.747+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:21:58.044 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:21:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:21:57.747+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:21:58.044 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:21:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:21:57.854+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:21:58.044 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:21:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:21:57.854+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:21:58.044 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:21:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:21:57.854+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:21:59.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:21:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:21:58.845+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:21:59.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:21:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:21:58.845+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:21:59.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:21:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:21:58.845+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:21:59.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:21:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:21:58.753+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:21:59.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:21:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:21:58.753+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:21:59.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:21:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:21:58.753+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:21:59.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:21:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:21:58.687+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:21:59.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:21:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:21:58.687+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:21:59.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:21:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:21:58.687+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:22:00.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:21:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:21:59.852+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:22:00.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:21:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:21:59.852+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:22:00.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:21:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:21:59.852+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:22:00.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:21:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:21:59.764+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:22:00.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:21:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:21:59.764+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:22:00.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:21:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:21:59.764+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:22:00.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:21:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:21:59.678+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:22:00.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:21:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:21:59.678+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:22:00.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:21:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:21:59.678+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:22:00.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:22:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:22:00.142+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 29 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:22:01.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:22:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:22:00.793+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:22:01.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:22:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:22:00.793+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:22:01.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:22:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:22:00.793+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:22:01.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:22:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:22:00.633+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:22:01.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:22:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:22:00.633+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:22:01.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:22:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:22:00.633+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:22:01.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:22:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:22:00.843+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:22:01.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:22:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:22:00.843+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:22:01.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:22:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:22:00.843+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:22:02.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:22:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:22:01.871+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:22:02.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:22:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:22:01.871+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:22:02.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:22:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:22:01.871+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:22:02.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:22:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:22:01.835+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:22:02.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:22:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:22:01.835+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:22:02.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:22:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:22:01.835+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:22:02.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:22:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:22:01.681+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:22:02.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:22:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:22:01.681+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:22:02.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:22:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:22:01.681+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:22:03.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:22:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:22:02.847+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:22:03.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:22:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:22:02.847+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:22:03.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:22:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:22:02.847+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:22:03.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:22:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:22:02.864+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:22:03.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:22:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:22:02.864+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:22:03.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:22:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:22:02.864+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:22:03.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:22:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:22:02.652+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:22:03.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:22:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:22:02.652+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:22:03.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:22:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:22:02.652+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:22:03.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:22:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:22:03.648+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:22:03.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:22:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:22:03.648+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:22:03.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:22:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:22:03.648+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:22:04.186 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:22:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:22:03.872+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:22:04.187 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:22:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:22:03.872+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:22:04.187 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:22:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:22:03.872+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:22:04.187 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:22:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:22:03.844+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:22:04.187 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:22:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:22:03.844+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:22:04.187 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:22:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:22:03.844+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:22:04.915 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:22:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:22:04.853+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:22:04.915 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:22:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:22:04.853+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:22:04.915 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:22:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:22:04.853+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:22:04.916 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:22:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:22:04.641+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:22:04.916 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:22:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:22:04.641+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:22:04.916 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:22:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:22:04.641+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:22:05.166 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:22:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:22:05.144+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 54 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:22:05.166 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:22:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:22:04.915+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:22:05.167 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:22:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:22:04.915+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:22:05.167 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:22:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:22:04.915+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:22:05.936 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:22:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:22:05.682+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:22:05.936 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:22:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:22:05.682+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:22:05.936 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:22:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:22:05.682+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:22:05.936 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:22:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:22:05.872+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:22:05.936 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:22:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:22:05.872+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:22:05.936 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:22:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:22:05.872+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:22:06.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:22:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:22:05.935+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:22:06.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:22:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:22:05.935+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:22:06.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:22:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:22:05.935+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:22:07.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:22:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:22:06.920+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:22:07.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:22:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:22:06.920+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:22:07.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:22:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:22:06.920+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:22:07.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:22:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:22:06.889+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:22:07.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:22:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:22:06.889+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:22:07.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:22:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:22:06.889+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:22:07.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:22:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:22:06.700+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:22:07.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:22:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:22:06.700+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:22:07.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:22:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:22:06.700+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:22:08.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:22:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:22:07.679+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:22:08.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:22:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:22:07.679+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:22:08.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:22:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:22:07.679+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:22:08.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:22:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:22:07.848+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:22:08.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:22:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:22:07.848+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:22:08.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:22:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:22:07.848+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:22:08.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:22:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:22:07.877+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:22:08.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:22:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:22:07.877+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:22:08.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:22:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:22:07.877+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:22:09.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:22:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:22:08.648+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:22:09.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:22:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:22:08.648+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:22:09.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:22:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:22:08.648+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:22:09.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:22:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:22:08.822+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:22:09.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:22:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:22:08.822+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:22:09.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:22:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:22:08.822+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:22:09.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:22:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:22:08.842+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:22:09.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:22:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:22:08.842+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:22:09.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:22:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:22:08.842+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:22:10.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:22:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:22:09.662+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:22:10.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:22:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:22:09.662+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:22:10.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:22:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:22:09.662+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:22:10.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:22:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:22:09.808+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:22:10.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:22:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:22:09.808+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:22:10.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:22:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:22:09.808+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:22:10.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:22:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:22:09.869+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:22:10.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:22:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:22:09.869+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:22:10.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:22:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:22:09.869+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:22:10.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:22:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:22:10.144+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 81 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:22:11.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:22:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:22:10.710+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:22:11.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:22:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:22:10.710+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:22:11.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:22:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:22:10.710+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:22:11.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:22:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:22:10.772+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:22:11.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:22:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:22:10.772+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:22:11.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:22:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:22:10.772+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:22:11.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:22:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:22:10.857+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:22:11.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:22:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:22:10.857+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:22:11.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:22:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:22:10.857+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:22:11.879 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:22:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:22:11.772+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:22:11.879 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:22:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:22:11.772+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:22:11.879 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:22:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:22:11.772+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:22:11.879 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:22:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:22:11.688+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:22:11.879 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:22:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:22:11.688+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:22:11.879 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:22:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:22:11.688+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:22:12.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:22:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:22:11.878+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:22:12.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:22:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:22:11.878+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:22:12.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:22:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:22:11.878+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:22:12.929 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:22:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:22:12.639+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:22:12.929 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:22:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:22:12.639+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:22:12.929 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:22:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:22:12.639+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:22:12.929 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:22:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:22:12.734+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:22:12.929 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:22:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:22:12.734+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:22:12.929 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:22:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:22:12.734+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:22:13.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:22:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:22:12.928+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:22:13.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:22:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:22:12.928+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:22:13.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:22:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:22:12.928+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:22:14.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:22:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:22:13.755+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:22:14.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:22:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:22:13.755+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:22:14.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:22:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:22:13.755+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:22:14.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:22:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:22:13.896+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:22:14.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:22:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:22:13.896+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:22:14.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:22:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:22:13.896+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:22:14.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:22:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:22:13.689+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:22:14.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:22:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:22:13.689+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:22:14.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:22:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:22:13.689+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:22:15.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:22:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:22:14.907+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:22:15.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:22:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:22:14.907+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:22:15.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:22:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:22:14.907+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:22:15.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:22:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:22:14.781+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:22:15.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:22:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:22:14.781+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:22:15.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:22:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:22:14.781+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:22:15.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:22:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:22:14.734+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:22:15.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:22:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:22:14.734+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:22:15.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:22:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:22:14.734+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:22:15.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:22:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:22:15.145+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 110 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:22:16.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:22:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:22:15.874+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:22:16.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:22:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:22:15.874+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:22:16.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:22:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:22:15.874+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:22:16.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:22:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:22:15.744+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:22:16.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:22:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:22:15.744+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:22:16.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:22:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:22:15.744+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:22:16.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:22:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:22:15.710+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:22:16.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:22:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:22:15.710+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:22:16.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:22:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:22:15.710+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:22:17.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:22:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:22:16.755+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:22:17.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:22:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:22:16.755+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:22:17.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:22:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:22:16.755+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:22:17.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:22:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:22:16.751+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:22:17.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:22:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:22:16.751+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:22:17.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:22:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:22:16.751+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:22:17.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:22:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:22:16.911+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:22:17.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:22:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:22:16.911+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:22:17.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:22:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:22:16.911+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:22:18.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:22:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:22:17.804+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:22:18.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:22:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:22:17.804+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:22:18.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:22:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:22:17.804+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:22:18.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:22:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:22:17.738+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:22:18.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:22:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:22:17.738+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:22:18.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:22:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:22:17.738+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:22:18.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:22:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:22:17.911+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:22:18.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:22:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:22:17.911+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:22:18.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:22:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:22:17.911+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:22:19.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:22:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:22:18.851+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:22:19.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:22:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:22:18.851+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:22:19.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:22:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:22:18.851+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:22:19.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:22:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:22:18.745+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:22:19.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:22:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:22:18.745+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:22:19.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:22:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:22:18.745+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:22:19.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:22:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:22:18.906+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:22:19.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:22:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:22:18.906+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:22:19.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:22:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:22:18.906+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:22:20.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:22:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:22:19.706+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:22:20.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:22:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:22:19.706+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:22:20.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:22:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:22:19.707+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:22:20.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:22:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:22:19.827+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:22:20.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:22:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:22:19.827+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:22:20.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:22:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:22:19.827+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:22:20.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:22:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:22:19.911+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:22:20.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:22:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:22:19.911+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:22:20.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:22:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:22:19.911+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:22:20.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:22:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:22:20.145+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 137 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:22:21.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:22:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:22:20.930+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:22:21.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:22:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:22:20.930+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:22:21.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:22:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:22:20.930+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:22:21.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:22:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:22:20.680+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:22:21.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:22:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:22:20.680+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:22:21.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:22:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:22:20.680+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:22:21.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:22:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:22:20.787+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:22:21.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:22:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:22:20.787+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:22:21.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:22:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:22:20.787+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:22:21.886 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:22:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:22:21.638+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:22:21.886 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:22:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:22:21.638+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:22:21.886 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:22:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:22:21.638+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:22:21.887 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:22:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:22:21.835+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:22:21.887 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:22:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:22:21.835+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:22:21.887 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:22:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:22:21.835+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:22:22.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:22:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:22:21.885+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:22:22.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:22:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:22:21.885+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:22:22.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:22:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:22:21.885+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:22:22.868 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:22:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:22:22.849+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:22:22.869 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:22:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:22:22.849+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:22:22.869 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:22:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:22:22.849+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:22:22.869 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:22:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:22:22.675+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:22:22.869 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:22:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:22:22.675+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:22:22.869 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:22:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:22:22.675+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:22:23.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:22:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:22:22.868+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:22:23.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:22:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:22:22.868+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:22:23.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:22:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:22:22.868+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:22:23.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:22:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:22:23.652+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:22:23.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:22:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:22:23.652+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:22:23.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:22:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:22:23.652+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:22:24.319 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:22:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:22:23.850+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:22:24.319 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:22:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:22:23.850+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:22:24.319 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:22:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:22:23.850+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:22:24.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:22:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:22:23.861+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:22:24.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:22:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:22:23.861+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:22:24.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:22:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:22:23.861+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:22:25.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:22:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:22:24.837+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:22:25.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:22:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:22:24.837+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:22:25.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:22:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:22:24.837+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:22:25.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:22:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:22:24.603+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:22:25.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:22:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:22:24.603+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:22:25.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:22:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:22:24.603+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:22:25.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:22:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:22:24.837+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:22:25.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:22:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:22:24.837+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:22:25.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:22:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:22:24.837+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:22:25.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:22:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:22:25.146+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 165 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:22:25.840 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:22:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:22:25.584+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:22:25.840 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:22:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:22:25.584+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:22:25.840 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:22:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:22:25.584+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:22:26.319 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:22:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:22:25.881+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:22:26.319 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:22:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:22:25.881+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:22:26.319 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:22:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:22:25.881+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:22:26.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:22:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:22:25.839+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:22:26.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:22:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:22:25.839+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:22:26.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:22:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:22:25.839+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:22:26.818 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:22:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:22:26.544+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:22:26.818 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:22:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:22:26.544+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:22:26.818 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:22:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:22:26.544+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:22:27.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:22:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:22:26.846+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:22:27.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:22:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:22:26.846+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:22:27.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:22:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:22:26.846+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:22:27.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:22:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:22:26.817+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:22:27.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:22:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:22:26.817+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:22:27.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:22:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:22:26.817+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:22:28.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:22:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:22:27.593+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:22:28.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:22:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:22:27.593+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:22:28.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:22:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:22:27.593+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:22:28.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:22:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:22:27.808+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:22:28.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:22:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:22:27.808+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:22:28.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:22:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:22:27.808+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:22:28.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:22:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:22:27.779+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:22:28.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:22:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:22:27.779+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:22:28.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:22:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:22:27.779+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:22:29.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:22:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:22:28.631+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:22:29.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:22:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:22:28.631+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:22:29.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:22:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:22:28.631+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:22:29.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:22:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:22:28.789+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:22:29.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:22:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:22:28.789+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:22:29.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:22:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:22:28.789+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:22:29.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:22:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:22:28.796+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:22:29.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:22:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:22:28.796+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:22:29.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:22:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:22:28.796+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:22:30.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:22:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:22:29.610+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:22:30.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:22:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:22:29.610+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:22:30.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:22:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:22:29.610+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:22:30.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:22:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:22:29.776+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:22:30.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:22:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:22:29.776+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:22:30.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:22:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:22:29.776+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:22:30.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:22:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:22:29.827+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:22:30.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:22:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:22:29.827+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:22:30.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:22:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:22:29.827+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:22:30.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:22:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:22:30.146+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 194 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:22:30.861 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:22:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:22:30.618+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:22:30.862 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:22:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:22:30.618+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:22:30.862 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:22:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:22:30.618+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:22:30.862 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:22:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:22:30.765+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:22:30.862 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:22:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:22:30.765+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:22:30.862 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:22:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:22:30.765+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:22:31.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:22:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:22:30.861+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:22:31.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:22:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:22:30.861+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:22:31.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:22:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:22:30.861+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:22:31.910 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:22:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:22:31.797+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:22:31.910 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:22:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:22:31.797+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:22:31.910 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:22:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:22:31.797+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:22:31.911 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:22:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:22:31.611+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:22:31.911 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:22:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:22:31.611+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:22:31.911 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:22:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:22:31.611+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:22:32.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:22:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:22:31.909+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:22:32.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:22:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:22:31.909+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:22:32.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:22:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:22:31.909+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:22:32.896 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:22:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:22:32.777+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:22:32.896 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:22:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:22:32.777+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:22:32.896 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:22:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:22:32.777+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:22:32.897 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:22:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:22:32.622+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:22:32.897 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:22:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:22:32.622+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:22:32.897 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:22:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:22:32.622+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:22:33.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:22:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:22:32.896+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:22:33.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:22:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:22:32.896+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:22:33.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:22:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:22:32.896+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:22:33.813 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:22:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:22:33.627+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:22:33.813 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:22:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:22:33.627+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:22:33.814 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:22:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:22:33.627+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:22:34.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:22:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:22:33.812+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:22:34.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:22:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:22:33.812+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:22:34.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:22:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:22:33.812+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:22:34.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:22:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:22:33.863+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:22:34.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:22:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:22:33.863+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:22:34.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:22:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:22:33.863+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:22:34.907 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:22:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:22:34.624+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:22:34.907 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:22:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:22:34.624+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:22:34.907 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:22:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:22:34.624+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:22:34.907 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:22:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:22:34.776+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:22:34.907 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:22:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:22:34.776+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:22:34.907 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:22:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:22:34.776+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:22:34.908 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:22:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:22:34.848+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:22:34.908 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:22:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:22:34.848+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:22:34.908 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:22:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:22:34.848+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:22:35.157 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:22:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:22:35.146+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 221 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:22:36.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:22:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:22:35.640+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:22:36.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:22:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:22:35.640+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:22:36.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:22:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:22:35.640+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:22:36.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:22:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:22:35.762+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:22:36.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:22:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:22:35.762+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:22:36.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:22:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:22:35.762+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:22:36.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:22:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:22:35.842+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:22:36.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:22:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:22:35.842+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:22:36.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:22:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:22:35.842+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:22:37.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:22:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:22:36.723+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:22:37.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:22:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:22:36.723+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:22:37.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:22:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:22:36.723+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:22:37.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:22:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:22:36.593+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:22:37.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:22:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:22:36.593+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:22:37.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:22:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:22:36.593+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:22:37.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:22:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:22:36.799+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:22:37.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:22:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:22:36.799+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:22:37.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:22:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:22:36.799+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:22:37.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:22:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:22:37.726+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:22:37.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:22:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:22:37.726+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:22:37.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:22:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:22:37.726+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:22:37.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:22:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:22:37.566+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:22:37.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:22:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:22:37.566+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:22:37.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:22:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:22:37.566+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:22:37.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:22:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:22:37.814+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:22:37.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:22:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:22:37.814+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:22:37.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:22:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:22:37.814+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:22:38.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:22:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:22:38.695+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:22:38.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:22:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:22:38.695+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:22:38.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:22:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:22:38.695+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:22:38.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:22:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:22:38.536+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:22:38.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:22:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:22:38.536+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:22:38.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:22:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:22:38.536+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:22:38.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:22:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:22:38.768+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:22:38.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:22:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:22:38.768+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:22:38.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:22:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:22:38.768+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:22:40.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:22:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:22:39.581+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:22:40.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:22:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:22:39.581+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:22:40.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:22:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:22:39.581+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:22:40.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:22:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:22:39.721+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:22:40.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:22:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:22:39.721+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:22:40.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:22:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:22:39.721+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:22:40.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:22:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:22:39.790+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:22:40.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:22:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:22:39.790+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:22:40.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:22:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:22:39.790+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:22:40.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:22:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:22:40.147+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 248 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:22:41.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:22:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:22:40.609+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:22:41.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:22:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:22:40.609+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:22:41.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:22:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:22:40.609+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:22:41.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:22:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:22:40.817+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:22:41.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:22:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:22:40.817+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:22:41.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:22:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:22:40.817+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:22:41.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:22:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:22:40.756+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:22:41.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:22:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:22:40.756+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:22:41.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:22:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:22:40.756+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:22:42.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:22:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:22:41.574+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:22:42.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:22:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:22:41.574+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:22:42.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:22:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:22:41.574+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:22:42.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:22:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:22:41.746+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:22:42.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:22:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:22:41.746+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:22:42.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:22:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:22:41.746+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:22:42.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:22:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:22:41.817+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:22:42.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:22:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:22:41.817+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:22:42.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:22:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:22:41.817+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:22:43.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:22:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:22:42.602+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:22:43.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:22:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:22:42.602+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:22:43.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:22:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:22:42.602+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:22:43.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:22:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:22:42.756+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:22:43.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:22:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:22:42.756+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:22:43.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:22:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:22:42.756+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:22:43.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:22:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:22:42.807+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:22:43.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:22:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:22:42.808+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:22:43.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:22:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:22:42.808+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:22:43.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:22:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:22:43.647+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:22:43.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:22:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:22:43.647+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:22:43.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:22:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:22:43.647+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:22:43.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:22:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:22:43.746+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:22:43.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:22:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:22:43.746+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:22:43.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:22:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:22:43.746+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:22:44.122 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:22:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:22:43.833+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:22:44.122 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:22:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:22:43.833+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:22:44.123 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:22:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:22:43.833+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:22:45.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:22:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:22:44.785+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:22:45.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:22:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:22:44.785+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:22:45.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:22:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:22:44.785+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:22:45.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:22:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:22:44.647+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:22:45.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:22:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:22:44.647+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:22:45.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:22:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:22:44.647+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:22:45.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:22:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:22:44.779+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:22:45.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:22:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:22:44.779+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:22:45.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:22:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:22:44.779+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:22:45.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:22:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:22:45.147+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 276 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:22:46.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:22:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:22:45.669+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:22:46.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:22:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:22:45.669+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:22:46.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:22:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:22:45.669+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:22:46.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:22:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:22:45.766+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:22:46.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:22:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:22:45.766+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:22:46.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:22:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:22:45.766+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:22:46.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:22:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:22:45.762+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:22:46.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:22:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:22:45.762+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:22:46.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:22:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:22:45.762+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:22:47.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:22:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:22:46.738+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:22:47.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:22:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:22:46.738+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:22:47.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:22:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:22:46.738+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:22:47.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:22:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:22:46.708+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:22:47.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:22:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:22:46.708+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:22:47.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:22:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:22:46.708+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:22:47.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:22:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:22:46.734+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:22:47.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:22:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:22:46.734+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:22:47.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:22:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:22:46.734+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:22:48.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:22:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:22:47.703+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:22:48.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:22:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:22:47.703+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:22:48.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:22:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:22:47.703+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:22:48.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:22:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:22:47.779+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:22:48.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:22:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:22:47.779+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:22:48.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:22:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:22:47.779+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:22:48.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:22:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:22:47.707+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:22:48.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:22:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:22:47.707+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:22:48.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:22:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:22:47.707+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:22:49.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:22:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:22:48.818+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:22:49.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:22:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:22:48.818+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:22:49.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:22:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:22:48.818+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:22:49.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:22:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:22:48.685+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:22:49.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:22:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:22:48.685+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:22:49.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:22:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:22:48.685+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:22:49.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:22:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:22:48.708+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:22:49.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:22:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:22:48.708+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:22:49.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:22:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:22:48.708+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:22:49.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:22:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:22:48.709+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14480.15:263 2.13 2:ceffbd45:::200.0000000a:head [write 1685963~1455 in=1455b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:22:50.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:22:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:22:49.750+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:22:50.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:22:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:22:49.750+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:22:50.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:22:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:22:49.750+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:22:50.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:22:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:22:49.750+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14480.15:263 2.13 2:ceffbd45:::200.0000000a:head [write 1685963~1455 in=1455b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:22:50.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:22:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:22:49.777+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:22:50.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:22:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:22:49.777+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:22:50.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:22:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:22:49.778+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:22:50.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:22:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:22:49.709+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:22:50.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:22:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:22:49.709+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:22:50.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:22:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:22:49.709+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:22:50.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:22:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:22:50.148+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 304 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:22:51.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:22:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:22:50.797+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:22:51.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:22:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:22:50.797+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:22:51.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:22:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:22:50.797+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:22:51.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:22:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:22:50.721+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:22:51.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:22:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:22:50.721+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:22:51.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:22:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:22:50.721+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:22:51.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:22:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:22:50.721+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14480.15:263 2.13 2:ceffbd45:::200.0000000a:head [write 1685963~1455 in=1455b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:22:51.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:22:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:22:50.676+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:22:51.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:22:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:22:50.676+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:22:51.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:22:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:22:50.676+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:22:52.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:22:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:22:51.792+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:22:52.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:22:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:22:51.792+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:22:52.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:22:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:22:51.792+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:22:52.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:22:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:22:51.710+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:22:52.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:22:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:22:51.710+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:22:52.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:22:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:22:51.710+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:22:52.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:22:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:22:51.710+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14480.15:263 2.13 2:ceffbd45:::200.0000000a:head [write 1685963~1455 in=1455b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:22:52.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:22:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:22:51.642+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:22:52.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:22:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:22:51.642+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:22:52.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:22:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:22:51.642+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:22:53.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:22:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:22:52.728+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:22:53.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:22:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:22:52.728+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:22:53.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:22:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:22:52.728+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:22:53.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:22:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:22:52.728+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14480.15:263 2.13 2:ceffbd45:::200.0000000a:head [write 1685963~1455 in=1455b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:22:53.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:22:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:22:52.807+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:22:53.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:22:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:22:52.807+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:22:53.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:22:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:22:52.807+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:22:53.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:22:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:22:52.643+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:22:53.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:22:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:22:52.643+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:22:53.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:22:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:22:52.643+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:22:54.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:22:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:22:53.827+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:22:54.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:22:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:22:53.827+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:22:54.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:22:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:22:53.827+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:22:54.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:22:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:22:53.601+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:22:54.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:22:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:22:53.601+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:22:54.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:22:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:22:53.601+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:22:54.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:22:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:22:53.689+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:22:54.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:22:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:22:53.689+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:22:54.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:22:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:22:53.689+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:22:54.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:22:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:22:53.689+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14480.15:263 2.13 2:ceffbd45:::200.0000000a:head [write 1685963~1455 in=1455b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:22:54.853 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:22:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:22:54.571+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:22:54.853 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:22:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:22:54.571+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:22:54.853 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:22:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:22:54.571+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:22:54.853 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:22:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:22:54.696+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:22:54.854 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:22:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:22:54.696+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:22:54.854 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:22:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:22:54.696+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:22:54.854 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:22:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:22:54.696+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14480.15:263 2.13 2:ceffbd45:::200.0000000a:head [write 1685963~1455 in=1455b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:22:55.149 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:22:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:22:54.851+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:22:55.149 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:22:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:22:54.851+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:22:55.149 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:22:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:22:54.851+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:22:55.542 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:22:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:22:55.147+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 331 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:22:55.813 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:22:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:22:55.540+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:22:55.813 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:22:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:22:55.540+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:22:55.814 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:22:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:22:55.540+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:22:55.814 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:22:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:22:55.541+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:22:55.814 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:22:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:22:55.724+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:22:55.814 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:22:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:22:55.724+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:22:55.814 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:22:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:22:55.724+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:22:55.814 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:22:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:22:55.726+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:22:56.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:22:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:22:55.812+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:22:56.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:22:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:22:55.812+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:22:56.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:22:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:22:55.812+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:22:56.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:22:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:22:55.813+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:22:56.704 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:22:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:22:56.512+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:22:56.704 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:22:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:22:56.512+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:22:56.704 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:22:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:22:56.512+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:22:56.704 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:22:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:22:56.513+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:22:57.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:22:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:22:56.702+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:22:57.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:22:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:22:56.702+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:22:57.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:22:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:22:56.702+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:22:57.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:22:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:22:56.703+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:22:57.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:22:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:22:56.853+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:22:57.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:22:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:22:56.853+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:22:57.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:22:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:22:56.853+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:22:57.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:22:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:22:56.854+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:22:57.814 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:22:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:22:57.660+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:22:57.814 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:22:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:22:57.660+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:22:57.814 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:22:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:22:57.660+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:22:57.814 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:22:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:22:57.661+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:22:57.815 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:22:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:22:57.537+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:22:57.815 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:22:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:22:57.537+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:22:57.815 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:22:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:22:57.537+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:22:57.815 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:22:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:22:57.538+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:22:58.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:22:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:22:57.812+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:22:58.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:22:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:22:57.812+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:22:58.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:22:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:22:57.812+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:22:58.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:22:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:22:57.813+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:22:59.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:22:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:22:58.792+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:22:59.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:22:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:22:58.792+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:22:59.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:22:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:22:58.792+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:22:59.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:22:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:22:58.793+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:22:59.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:22:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:22:58.668+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:22:59.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:22:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:22:58.668+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:22:59.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:22:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:22:58.668+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:22:59.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:22:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:22:58.668+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:22:59.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:22:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:22:58.574+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:22:59.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:22:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:22:58.574+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:22:59.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:22:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:22:58.574+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:22:59.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:22:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:22:58.574+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:23:00.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:22:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:22:59.793+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:23:00.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:22:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:22:59.793+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:23:00.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:22:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:22:59.793+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:23:00.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:22:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:22:59.793+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:23:00.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:22:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:22:59.596+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:23:00.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:22:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:22:59.596+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:23:00.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:22:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:22:59.596+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:23:00.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:22:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:22:59.597+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:23:00.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:22:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:22:59.674+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:23:00.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:22:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:22:59.674+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:23:00.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:22:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:22:59.674+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:23:00.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:22:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:22:59.675+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:23:00.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:23:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:23:00.148+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 360 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:23:01.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:23:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:23:00.769+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:23:01.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:23:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:23:00.769+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:23:01.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:23:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:23:00.769+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:23:01.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:23:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:23:00.770+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:23:01.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:23:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:23:00.618+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:23:01.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:23:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:23:00.618+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:23:01.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:23:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:23:00.618+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:23:01.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:23:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:23:00.619+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:23:01.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:23:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:23:00.654+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:23:01.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:23:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:23:00.654+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:23:01.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:23:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:23:00.654+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:23:01.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:23:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:23:00.655+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:23:02.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:23:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:23:01.813+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:23:02.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:23:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:23:01.813+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:23:02.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:23:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:23:01.813+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:23:02.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:23:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:23:01.814+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:23:02.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:23:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:23:01.670+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:23:02.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:23:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:23:01.670+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:23:02.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:23:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:23:01.670+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:23:02.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:23:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:23:01.671+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:23:02.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:23:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:23:01.587+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:23:02.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:23:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:23:01.587+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:23:02.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:23:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:23:01.587+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:23:02.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:23:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:23:01.588+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:23:02.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:23:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:23:02.673+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:23:02.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:23:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:23:02.673+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:23:02.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:23:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:23:02.673+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:23:02.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:23:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:23:02.673+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:23:02.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:23:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:23:02.615+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:23:02.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:23:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:23:02.615+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:23:02.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:23:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:23:02.615+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:23:02.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:23:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:23:02.615+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:23:03.319 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:23:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:23:02.842+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:23:03.319 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:23:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:23:02.842+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:23:03.319 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:23:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:23:02.842+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:23:03.319 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:23:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:23:02.843+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:23:03.815 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:23:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:23:03.591+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:23:03.815 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:23:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:23:03.591+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:23:03.815 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:23:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:23:03.591+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:23:03.815 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:23:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:23:03.591+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:23:03.815 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:23:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:23:03.655+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:23:03.815 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:23:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:23:03.655+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:23:03.815 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:23:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:23:03.655+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:23:03.815 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:23:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:23:03.655+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:23:04.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:23:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:23:03.814+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:23:04.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:23:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:23:03.814+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:23:04.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:23:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:23:03.814+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:23:04.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:23:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:23:03.814+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:23:04.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:23:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:23:04.650+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:23:04.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:23:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:23:04.650+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:23:04.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:23:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:23:04.650+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:23:04.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:23:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:23:04.651+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:23:04.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:23:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:23:04.560+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:23:04.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:23:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:23:04.560+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:23:04.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:23:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:23:04.560+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:23:04.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:23:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:23:04.560+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:23:04.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:23:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:23:04.785+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:23:04.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:23:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:23:04.785+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:23:04.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:23:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:23:04.785+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:23:04.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:23:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:23:04.786+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:23:05.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:23:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:23:05.149+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 388 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:23:06.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:23:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:23:05.666+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:23:06.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:23:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:23:05.666+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:23:06.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:23:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:23:05.666+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:23:06.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:23:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:23:05.667+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:23:06.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:23:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:23:05.589+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:23:06.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:23:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:23:05.589+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:23:06.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:23:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:23:05.589+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:23:06.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:23:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:23:05.589+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:23:06.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:23:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:23:05.750+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:23:06.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:23:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:23:05.750+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:23:06.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:23:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:23:05.750+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:23:06.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:23:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:23:05.751+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:23:06.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:23:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:23:06.708+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:23:06.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:23:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:23:06.708+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:23:06.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:23:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:23:06.708+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:23:06.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:23:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:23:06.708+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:23:06.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:23:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:23:06.540+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:23:06.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:23:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:23:06.540+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:23:06.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:23:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:23:06.540+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:23:06.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:23:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:23:06.540+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:23:06.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:23:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:23:06.764+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:23:06.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:23:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:23:06.764+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:23:06.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:23:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:23:06.764+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:23:06.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:23:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:23:06.765+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:23:07.796 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:23:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:23:07.538+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:23:07.796 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:23:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:23:07.538+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:23:07.796 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:23:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:23:07.538+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:23:07.796 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:23:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:23:07.538+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:23:07.796 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:23:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:23:07.738+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:23:07.796 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:23:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:23:07.738+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:23:07.796 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:23:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:23:07.738+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:23:07.797 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:23:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:23:07.739+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:23:08.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:23:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:23:07.795+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:23:08.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:23:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:23:07.795+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:23:08.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:23:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:23:07.795+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:23:08.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:23:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:23:07.796+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:23:08.751 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:23:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:23:08.495+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:23:08.751 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:23:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:23:08.495+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:23:08.751 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:23:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:23:08.495+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:23:08.751 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:23:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:23:08.496+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:23:09.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:23:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:23:08.771+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:23:09.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:23:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:23:08.771+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:23:09.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:23:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:23:08.771+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:23:09.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:23:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:23:08.772+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:23:09.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:23:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:23:08.750+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:23:09.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:23:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:23:08.750+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:23:09.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:23:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:23:08.750+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:23:09.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:23:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:23:08.751+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:23:09.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:23:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:23:09.506+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:23:09.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:23:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:23:09.506+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:23:09.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:23:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:23:09.506+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:23:09.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:23:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:23:09.506+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:23:09.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:23:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:23:09.740+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:23:09.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:23:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:23:09.740+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:23:09.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:23:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:23:09.740+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:23:09.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:23:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:23:09.741+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:23:09.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:23:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:23:09.734+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:23:09.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:23:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:23:09.734+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:23:09.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:23:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:23:09.734+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:23:09.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:23:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:23:09.735+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:23:10.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:23:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:23:10.150+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 416 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:23:10.763 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:23:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:23:10.458+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:23:10.763 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:23:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:23:10.458+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:23:10.763 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:23:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:23:10.458+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:23:10.763 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:23:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:23:10.458+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:23:10.764 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:23:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:23:10.687+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:23:10.764 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:23:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:23:10.687+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:23:10.764 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:23:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:23:10.687+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:23:10.764 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:23:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:23:10.688+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:23:11.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:23:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:23:10.762+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:23:11.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:23:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:23:10.762+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:23:11.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:23:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:23:10.762+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:23:11.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:23:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:23:10.764+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:23:11.770 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:23:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:23:11.502+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:23:11.771 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:23:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:23:11.502+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:23:11.771 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:23:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:23:11.502+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:23:11.771 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:23:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:23:11.502+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:23:11.771 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:23:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:23:11.742+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:23:11.771 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:23:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:23:11.742+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:23:11.771 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:23:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:23:11.742+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:23:11.771 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:23:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:23:11.743+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:23:11.771 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:23:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:23:11.647+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:23:11.771 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:23:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:23:11.647+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:23:11.772 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:23:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:23:11.647+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:23:11.772 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:23:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:23:11.648+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:23:12.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:23:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:23:12.734+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:23:12.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:23:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:23:12.734+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:23:12.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:23:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:23:12.734+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:23:12.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:23:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:23:12.734+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:23:12.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:23:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:23:12.530+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:23:12.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:23:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:23:12.530+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:23:12.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:23:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:23:12.530+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:23:12.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:23:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:23:12.531+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:23:12.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:23:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:23:12.648+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:23:12.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:23:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:23:12.648+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:23:12.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:23:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:23:12.648+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:23:12.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:23:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:23:12.649+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:23:13.778 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:23:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:23:13.616+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:23:13.779 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:23:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:23:13.616+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:23:13.779 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:23:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:23:13.616+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:23:13.779 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:23:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:23:13.617+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:23:13.779 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:23:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:23:13.487+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:23:13.779 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:23:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:23:13.487+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:23:13.779 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:23:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:23:13.487+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:23:13.779 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:23:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:23:13.488+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:23:14.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:23:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:23:13.778+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:23:14.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:23:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:23:13.778+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:23:14.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:23:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:23:13.778+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:23:14.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:23:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:23:13.778+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:23:14.753 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:23:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:23:14.598+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:23:14.753 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:23:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:23:14.598+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:23:14.753 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:23:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:23:14.598+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:23:14.754 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:23:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:23:14.598+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:23:14.754 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:23:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:23:14.452+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:23:14.754 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:23:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:23:14.452+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:23:14.754 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:23:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:23:14.452+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:23:14.754 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:23:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:23:14.452+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:23:14.754 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:23:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:23:14.752+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:23:14.754 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:23:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:23:14.752+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:23:14.754 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:23:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:23:14.752+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:23:15.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:23:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:23:14.753+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:23:15.470 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:23:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:23:15.150+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 446 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:23:15.721 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:23:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:23:15.470+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:23:15.721 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:23:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:23:15.470+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:23:15.721 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:23:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:23:15.470+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:23:15.721 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:23:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:23:15.470+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:23:15.721 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:23:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:23:15.572+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:23:15.722 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:23:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:23:15.572+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:23:15.722 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:23:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:23:15.572+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:23:15.722 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:23:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:23:15.573+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:23:16.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:23:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:23:15.720+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:23:16.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:23:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:23:15.720+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:23:16.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:23:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:23:15.720+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:23:16.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:23:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:23:15.721+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:23:16.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:23:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:23:16.596+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:23:16.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:23:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:23:16.596+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:23:16.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:23:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:23:16.596+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:23:16.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:23:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:23:16.597+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:23:16.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:23:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:23:16.682+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:23:16.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:23:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:23:16.682+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:23:16.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:23:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:23:16.682+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:23:16.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:23:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:23:16.682+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:23:16.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:23:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:23:16.459+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:23:16.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:23:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:23:16.459+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:23:16.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:23:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:23:16.459+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:23:16.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:23:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:23:16.460+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:23:17.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:23:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:23:17.568+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:23:17.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:23:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:23:17.568+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:23:17.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:23:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:23:17.568+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:23:17.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:23:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:23:17.568+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:23:17.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:23:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:23:17.703+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:23:17.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:23:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:23:17.703+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:23:17.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:23:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:23:17.703+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:23:17.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:23:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:23:17.704+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:23:17.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:23:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:23:17.455+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:23:17.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:23:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:23:17.455+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:23:17.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:23:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:23:17.455+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:23:17.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:23:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:23:17.455+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:23:18.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:23:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:23:18.540+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:23:18.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:23:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:23:18.540+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:23:18.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:23:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:23:18.540+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:23:18.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:23:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:23:18.540+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:23:18.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:23:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:23:18.690+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:23:18.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:23:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:23:18.690+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:23:18.821 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:23:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:23:18.690+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:23:18.821 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:23:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:23:18.690+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:23:18.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:23:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:23:18.467+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:23:18.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:23:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:23:18.467+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:23:18.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:23:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:23:18.467+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:23:18.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:23:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:23:18.468+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:23:19.724 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:23:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:23:19.585+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:23:19.724 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:23:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:23:19.585+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:23:19.724 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:23:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:23:19.585+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:23:19.724 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:23:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:23:19.586+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:23:19.724 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:23:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:23:19.469+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:23:19.724 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:23:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:23:19.469+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:23:19.725 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:23:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:23:19.469+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:23:19.725 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:23:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:23:19.470+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:23:20.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:23:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:23:19.723+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:23:20.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:23:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:23:19.723+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:23:20.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:23:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:23:19.723+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:23:20.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:23:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:23:19.724+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:23:20.509 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:23:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:23:20.150+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 476 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:23:20.764 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:23:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:23:20.587+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:23:20.764 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:23:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:23:20.587+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:23:20.764 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:23:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:23:20.587+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:23:20.764 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:23:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:23:20.588+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:23:20.764 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:23:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:23:20.508+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:23:20.765 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:23:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:23:20.508+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:23:20.765 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:23:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:23:20.508+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:23:20.765 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:23:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:23:20.509+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:23:21.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:23:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:23:20.763+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:23:21.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:23:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:23:20.763+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:23:21.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:23:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:23:20.763+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:23:21.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:23:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:23:20.764+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:23:21.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:23:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:23:21.553+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:23:21.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:23:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:23:21.553+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:23:21.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:23:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:23:21.553+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:23:21.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:23:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:23:21.554+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:23:21.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:23:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:23:21.755+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:23:21.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:23:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:23:21.755+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:23:21.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:23:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:23:21.755+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:23:21.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:23:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:23:21.755+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:23:21.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:23:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:23:21.520+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:23:21.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:23:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:23:21.520+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:23:21.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:23:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:23:21.520+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:23:21.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:23:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:23:21.520+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:23:22.778 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:23:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:23:22.576+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:23:22.779 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:23:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:23:22.577+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:23:22.779 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:23:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:23:22.577+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:23:22.779 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:23:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:23:22.578+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:23:22.779 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:23:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:23:22.476+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:23:22.779 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:23:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:23:22.476+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:23:22.779 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:23:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:23:22.476+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:23:22.779 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:23:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:23:22.477+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:23:23.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:23:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:23:22.778+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:23:23.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:23:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:23:22.778+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:23:23.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:23:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:23:22.778+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:23:23.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:23:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:23:22.778+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:23:23.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:23:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:23:23.565+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:23:23.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:23:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:23:23.565+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:23:23.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:23:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:23:23.565+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:23:23.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:23:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:23:23.565+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:23:23.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:23:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:23:23.507+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:23:23.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:23:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:23:23.507+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:23:23.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:23:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:23:23.507+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:23:23.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:23:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:23:23.507+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:23:23.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:23:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:23:23.744+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:23:23.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:23:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:23:23.744+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:23:23.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:23:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:23:23.744+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:23:23.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:23:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:23:23.745+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:23:24.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:23:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:23:24.536+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:23:24.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:23:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:23:24.536+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:23:24.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:23:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:23:24.536+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:23:24.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:23:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:23:24.537+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:23:24.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:23:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:23:24.724+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:23:24.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:23:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:23:24.724+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:23:24.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:23:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:23:24.724+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:23:24.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:23:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:23:24.724+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:23:24.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:23:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:23:24.530+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:23:24.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:23:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:23:24.530+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:23:24.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:23:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:23:24.530+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:23:24.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:23:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:23:24.530+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:23:25.503 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:23:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:23:25.151+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 504 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:23:25.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:23:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:23:25.503+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:23:25.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:23:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:23:25.503+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:23:25.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:23:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:23:25.503+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:23:25.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:23:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:23:25.503+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:23:25.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:23:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:23:25.708+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:23:25.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:23:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:23:25.708+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:23:25.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:23:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:23:25.708+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:23:25.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:23:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:23:25.709+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:23:25.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:23:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:23:25.544+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:23:25.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:23:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:23:25.544+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:23:25.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:23:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:23:25.544+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:23:25.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:23:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:23:25.544+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:23:26.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:23:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:23:26.544+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:23:26.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:23:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:23:26.544+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:23:26.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:23:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:23:26.544+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:23:26.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:23:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:23:26.544+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:23:26.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:23:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:23:26.748+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:23:26.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:23:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:23:26.748+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:23:26.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:23:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:23:26.748+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:23:26.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:23:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:23:26.749+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:23:26.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:23:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:23:26.573+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:23:26.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:23:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:23:26.573+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:23:26.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:23:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:23:26.573+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:23:26.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:23:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:23:26.574+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:23:27.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:23:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:23:27.700+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:23:27.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:23:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:23:27.700+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:23:27.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:23:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:23:27.700+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:23:27.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:23:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:23:27.701+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:23:27.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:23:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:23:27.530+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:23:27.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:23:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:23:27.530+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:23:27.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:23:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:23:27.530+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:23:27.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:23:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:23:27.531+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:23:27.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:23:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:23:27.591+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:23:27.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:23:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:23:27.591+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:23:27.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:23:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:23:27.591+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:23:27.821 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:23:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:23:27.592+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:23:28.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:23:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:23:28.612+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:23:28.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:23:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:23:28.612+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:23:28.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:23:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:23:28.612+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:23:28.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:23:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:23:28.612+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:23:28.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:23:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:23:28.746+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:23:28.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:23:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:23:28.746+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:23:28.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:23:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:23:28.746+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:23:28.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:23:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:23:28.747+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:23:28.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:23:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:23:28.547+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:23:28.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:23:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:23:28.547+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:23:28.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:23:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:23:28.547+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:23:28.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:23:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:23:28.548+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:23:29.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:23:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:23:29.748+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:23:29.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:23:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:23:29.748+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:23:29.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:23:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:23:29.748+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:23:29.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:23:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:23:29.749+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:23:29.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:23:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:23:29.514+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:23:29.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:23:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:23:29.514+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:23:29.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:23:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:23:29.514+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:23:29.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:23:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:23:29.514+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:23:29.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:23:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:23:29.627+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:23:29.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:23:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:23:29.627+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:23:29.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:23:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:23:29.627+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:23:29.821 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:23:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:23:29.628+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:23:30.512 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:23:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:23:30.151+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 535 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:23:30.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:23:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:23:30.512+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:23:30.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:23:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:23:30.512+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:23:30.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:23:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:23:30.512+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:23:30.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:23:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:23:30.512+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:23:30.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:23:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:23:30.757+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:23:30.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:23:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:23:30.757+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:23:30.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:23:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:23:30.757+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:23:30.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:23:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:23:30.758+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:23:30.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:23:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:23:30.616+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:23:30.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:23:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:23:30.616+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:23:30.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:23:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:23:30.616+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:23:30.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:23:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:23:30.617+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:23:31.764 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:23:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:23:31.513+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:23:31.764 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:23:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:23:31.513+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:23:31.764 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:23:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:23:31.513+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:23:31.764 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:23:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:23:31.514+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:23:31.765 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:23:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:23:31.611+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:23:31.765 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:23:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:23:31.611+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:23:31.765 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:23:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:23:31.611+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:23:31.765 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:23:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:23:31.612+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:23:32.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:23:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:23:31.763+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:23:32.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:23:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:23:31.763+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:23:32.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:23:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:23:31.763+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:23:32.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:23:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:23:31.764+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:23:32.782 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:23:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:23:32.590+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:23:32.806 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:23:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:23:32.590+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:23:32.806 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:23:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:23:32.590+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:23:32.806 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:23:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:23:32.591+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:23:32.806 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:23:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:23:32.482+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:23:32.806 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:23:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:23:32.482+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:23:32.806 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:23:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:23:32.482+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:23:32.807 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:23:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:23:32.482+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:23:33.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:23:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:23:32.782+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:23:33.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:23:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:23:32.782+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:23:33.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:23:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:23:32.782+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:23:33.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:23:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:23:32.783+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:23:33.569 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:23:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:23:33.500+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:23:33.569 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:23:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:23:33.500+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:23:33.569 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:23:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:23:33.500+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:23:33.569 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:23:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:23:33.500+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:23:34.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:23:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:23:33.819+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:23:34.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:23:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:23:33.819+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:23:34.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:23:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:23:33.819+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:23:34.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:23:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:23:33.820+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:23:34.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:23:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:23:33.620+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:23:34.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:23:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:23:33.620+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:23:34.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:23:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:23:33.620+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:23:34.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:23:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:23:33.621+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:23:34.810 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:23:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:23:34.617+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:23:34.810 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:23:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:23:34.617+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:23:34.811 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:23:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:23:34.617+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:23:34.811 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:23:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:23:34.618+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:23:34.811 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:23:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:23:34.529+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:23:34.811 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:23:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:23:34.529+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:23:34.811 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:23:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:23:34.529+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:23:34.811 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:23:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:23:34.530+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:23:35.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:23:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:23:34.809+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:23:35.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:23:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:23:34.809+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:23:35.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:23:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:23:34.809+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:23:35.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:23:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:23:34.810+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:23:35.489 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:23:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:23:35.152+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 565 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:23:35.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:23:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:23:35.595+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:23:35.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:23:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:23:35.595+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:23:35.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:23:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:23:35.595+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:23:35.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:23:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:23:35.596+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:23:35.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:23:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:23:35.488+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:23:35.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:23:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:23:35.488+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:23:35.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:23:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:23:35.489+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:23:35.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:23:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:23:35.489+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:23:36.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:23:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:23:35.833+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:23:36.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:23:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:23:35.833+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:23:36.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:23:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:23:35.833+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:23:36.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:23:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:23:35.834+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:23:36.797 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:23:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:23:36.563+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:23:36.797 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:23:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:23:36.563+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:23:36.797 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:23:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:23:36.563+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:23:36.797 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:23:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:23:36.563+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:23:36.797 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:23:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:23:36.452+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:23:36.797 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:23:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:23:36.452+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:23:36.798 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:23:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:23:36.452+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:23:36.798 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:23:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:23:36.452+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:23:37.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:23:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:23:36.796+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:23:37.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:23:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:23:36.796+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:23:37.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:23:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:23:36.796+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:23:37.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:23:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:23:36.797+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:23:37.773 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:23:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:23:37.421+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:23:37.774 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:23:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:23:37.421+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:23:37.774 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:23:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:23:37.421+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:23:37.774 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:23:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:23:37.421+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:23:37.774 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:23:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:23:37.576+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:23:37.774 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:23:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:23:37.576+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:23:37.774 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:23:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:23:37.576+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:23:37.774 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:23:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:23:37.578+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:23:38.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:23:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:23:37.773+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:23:38.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:23:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:23:37.773+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:23:38.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:23:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:23:37.773+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:23:38.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:23:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:23:37.773+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:23:38.778 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:23:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:23:38.600+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:23:38.778 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:23:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:23:38.600+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:23:38.778 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:23:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:23:38.600+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:23:38.778 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:23:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:23:38.601+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:23:38.778 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:23:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:23:38.386+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:23:38.779 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:23:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:23:38.386+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:23:38.779 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:23:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:23:38.386+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:23:38.779 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:23:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:23:38.386+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:23:39.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:23:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:23:38.777+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:23:39.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:23:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:23:38.777+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:23:39.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:23:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:23:38.777+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:23:39.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:23:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:23:38.778+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:23:39.569 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:23:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:23:39.351+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:23:39.569 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:23:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:23:39.351+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:23:39.569 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:23:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:23:39.351+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:23:39.569 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:23:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:23:39.351+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:23:40.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:23:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:23:39.766+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:23:40.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:23:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:23:39.766+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:23:40.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:23:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:23:39.766+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:23:40.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:23:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:23:39.767+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:23:40.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:23:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:23:39.640+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:23:40.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:23:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:23:39.640+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:23:40.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:23:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:23:39.640+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:23:40.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:23:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:23:39.641+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:23:40.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:23:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:23:40.152+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 592 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:23:40.569 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:23:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:23:40.361+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:23:40.569 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:23:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:23:40.361+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:23:40.569 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:23:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:23:40.361+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:23:40.569 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:23:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:23:40.362+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:23:41.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:23:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:23:40.797+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:23:41.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:23:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:23:40.797+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:23:41.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:23:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:23:40.797+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:23:41.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:23:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:23:40.798+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:23:41.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:23:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:23:40.675+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:23:41.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:23:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:23:40.675+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:23:41.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:23:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:23:40.675+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:23:41.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:23:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:23:40.675+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:23:41.569 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:23:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:23:41.314+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:23:41.569 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:23:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:23:41.314+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:23:41.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:23:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:23:41.314+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:23:41.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:23:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:23:41.314+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:23:42.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:23:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:23:41.845+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:23:42.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:23:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:23:41.845+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:23:42.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:23:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:23:41.845+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:23:42.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:23:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:23:41.846+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:23:42.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:23:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:23:41.661+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:23:42.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:23:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:23:41.661+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:23:42.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:23:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:23:41.661+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:23:42.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:23:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:23:41.662+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:23:42.645 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:23:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:23:42.361+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:23:42.646 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:23:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:23:42.361+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:23:42.646 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:23:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:23:42.361+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:23:42.646 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:23:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:23:42.362+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:23:43.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:23:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:23:42.645+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:23:43.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:23:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:23:42.645+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:23:43.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:23:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:23:42.645+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:23:43.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:23:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:23:42.645+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:23:43.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:23:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:23:42.856+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:23:43.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:23:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:23:42.856+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:23:43.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:23:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:23:42.856+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:23:43.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:23:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:23:42.856+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:23:43.680 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:23:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:23:43.408+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:23:43.680 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:23:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:23:43.408+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:23:43.681 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:23:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:23:43.408+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:23:43.681 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:23:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:23:43.408+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:23:44.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:23:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:23:43.680+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:23:44.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:23:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:23:43.680+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:23:44.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:23:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:23:43.680+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:23:44.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:23:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:23:43.681+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:23:44.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:23:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:23:43.885+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:23:44.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:23:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:23:43.885+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:23:44.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:23:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:23:43.885+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:23:44.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:23:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:23:43.886+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:23:44.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:23:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:23:44.635+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:23:44.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:23:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:23:44.635+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:23:44.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:23:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:23:44.635+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:23:44.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:23:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:23:44.636+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:23:44.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:23:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:23:44.417+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:23:44.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:23:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:23:44.417+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:23:44.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:23:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:23:44.417+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:23:44.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:23:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:23:44.418+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:23:45.154 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:23:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:23:44.876+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:23:45.154 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:23:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:23:44.876+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:23:45.154 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:23:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:23:44.876+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:23:45.154 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:23:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:23:44.877+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:23:45.447 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:23:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:23:45.153+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 620 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:23:45.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:23:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:23:45.446+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:23:45.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:23:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:23:45.446+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:23:45.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:23:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:23:45.446+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:23:45.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:23:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:23:45.447+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:23:45.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:23:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:23:45.640+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:23:45.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:23:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:23:45.640+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:23:45.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:23:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:23:45.640+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:23:45.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:23:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:23:45.641+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:23:46.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:23:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:23:45.870+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:23:46.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:23:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:23:45.870+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:23:46.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:23:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:23:45.870+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:23:46.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:23:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:23:45.870+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:23:46.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:23:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:23:46.595+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:23:46.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:23:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:23:46.595+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:23:46.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:23:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:23:46.595+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:23:46.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:23:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:23:46.595+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:23:46.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:23:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:23:46.491+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:23:46.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:23:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:23:46.491+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:23:46.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:23:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:23:46.491+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:23:46.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:23:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:23:46.491+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:23:47.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:23:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:23:46.860+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:23:47.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:23:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:23:46.860+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:23:47.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:23:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:23:46.860+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:23:47.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:23:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:23:46.861+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:23:47.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:23:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:23:47.478+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:23:47.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:23:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:23:47.478+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:23:47.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:23:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:23:47.478+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:23:47.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:23:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:23:47.478+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:23:47.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:23:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:23:47.642+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:23:47.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:23:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:23:47.642+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:23:47.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:23:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:23:47.642+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:23:47.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:23:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:23:47.642+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:23:48.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:23:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:23:47.827+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:23:48.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:23:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:23:47.827+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:23:48.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:23:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:23:47.827+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:23:48.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:23:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:23:47.828+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:23:48.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:23:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:23:48.656+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:23:48.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:23:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:23:48.656+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:23:48.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:23:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:23:48.656+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:23:48.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:23:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:23:48.656+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:23:48.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:23:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:23:48.454+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:23:48.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:23:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:23:48.454+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:23:48.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:23:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:23:48.454+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:23:48.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:23:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:23:48.454+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:23:49.115 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:23:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:23:48.841+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:23:49.116 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:23:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:23:48.841+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:23:49.116 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:23:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:23:48.841+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:23:49.116 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:23:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:23:48.842+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:23:49.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:23:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:23:49.478+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:23:49.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:23:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:23:49.478+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:23:49.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:23:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:23:49.478+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:23:49.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:23:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:23:49.478+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:23:49.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:23:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:23:49.655+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:23:49.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:23:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:23:49.655+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:23:49.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:23:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:23:49.655+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:23:49.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:23:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:23:49.655+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:23:50.154 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:23:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:23:49.876+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:23:50.154 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:23:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:23:49.876+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:23:50.154 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:23:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:23:49.876+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:23:50.155 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:23:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:23:49.877+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:23:50.502 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:23:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:23:50.153+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 648 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:23:50.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:23:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:23:50.692+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:23:50.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:23:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:23:50.692+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:23:50.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:23:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:23:50.692+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:23:50.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:23:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:23:50.693+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:23:50.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:23:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:23:50.501+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:23:50.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:23:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:23:50.501+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:23:50.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:23:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:23:50.501+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:23:50.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:23:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:23:50.502+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:23:51.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:23:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:23:50.833+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:23:51.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:23:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:23:50.833+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:23:51.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:23:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:23:50.833+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:23:51.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:23:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:23:50.834+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:23:51.817 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:23:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:23:51.722+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:23:51.817 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:23:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:23:51.722+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:23:51.817 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:23:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:23:51.722+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:23:51.817 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:23:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:23:51.724+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:23:51.817 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:23:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:23:51.494+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:23:51.817 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:23:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:23:51.494+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:23:51.817 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:23:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:23:51.494+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:23:51.818 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:23:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:23:51.494+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:23:52.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:23:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:23:51.816+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:23:52.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:23:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:23:51.816+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:23:52.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:23:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:23:51.816+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:23:52.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:23:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:23:51.817+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:23:52.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:23:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:23:52.464+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:23:52.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:23:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:23:52.464+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:23:52.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:23:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:23:52.464+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:23:52.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:23:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:23:52.465+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:23:53.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:23:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:23:52.725+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:23:53.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:23:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:23:52.725+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:23:53.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:23:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:23:52.725+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:23:53.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:23:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:23:52.726+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:23:53.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:23:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:23:52.855+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:23:53.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:23:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:23:52.855+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:23:53.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:23:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:23:52.855+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:23:53.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:23:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:23:52.855+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:23:53.760 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:23:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:23:53.446+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:23:53.760 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:23:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:23:53.446+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:23:53.761 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:23:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:23:53.446+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:23:53.761 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:23:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:23:53.446+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:23:54.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:23:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:23:53.759+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:23:54.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:23:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:23:53.759+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:23:54.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:23:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:23:53.759+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:23:54.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:23:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:23:53.759+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:23:54.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:23:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:23:53.850+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:23:54.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:23:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:23:53.850+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:23:54.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:23:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:23:53.850+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:23:54.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:23:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:23:53.851+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:23:54.741 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:23:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:23:54.444+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:23:54.741 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:23:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:23:54.444+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:23:54.741 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:23:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:23:54.444+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:23:54.741 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:23:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:23:54.445+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:23:55.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:23:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:23:54.739+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:23:55.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:23:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:23:54.739+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:23:55.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:23:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:23:54.739+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:23:55.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:23:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:23:54.740+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:23:55.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:23:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:23:54.816+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:23:55.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:23:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:23:54.816+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:23:55.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:23:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:23:54.816+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:23:55.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:23:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:23:54.817+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:23:55.437 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:23:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:23:55.153+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 675 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:23:55.703 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:23:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:23:55.435+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:23:55.703 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:23:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:23:55.435+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:23:55.703 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:23:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:23:55.435+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:23:55.703 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:23:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:23:55.435+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:23:56.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:23:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:23:55.789+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:23:56.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:23:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:23:55.789+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:23:56.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:23:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:23:55.789+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:23:56.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:23:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:23:55.790+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:23:56.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:23:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:23:55.701+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:23:56.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:23:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:23:55.701+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:23:56.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:23:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:23:55.701+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:23:56.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:23:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:23:55.702+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:23:56.778 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:23:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:23:56.677+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:23:56.778 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:23:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:23:56.677+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:23:56.778 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:23:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:23:56.677+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:23:56.778 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:23:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:23:56.678+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:23:56.778 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:23:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:23:56.428+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:23:56.778 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:23:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:23:56.429+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:23:56.779 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:23:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:23:56.429+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:23:56.779 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:23:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:23:56.429+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:23:57.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:23:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:23:56.776+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:23:57.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:23:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:23:56.776+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:23:57.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:23:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:23:56.776+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:23:57.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:23:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:23:56.777+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:23:57.733 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:23:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:23:57.637+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:23:57.733 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:23:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:23:57.637+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:23:57.733 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:23:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:23:57.637+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:23:57.733 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:23:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:23:57.638+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:23:57.733 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:23:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:23:57.446+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:23:57.734 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:23:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:23:57.446+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:23:57.734 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:23:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:23:57.446+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:23:57.734 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:23:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:23:57.446+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:23:58.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:23:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:23:57.731+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:23:58.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:23:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:23:57.731+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:23:58.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:23:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:23:57.731+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:23:58.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:23:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:23:57.732+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:23:58.724 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:23:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:23:58.618+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:23:58.725 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:23:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:23:58.618+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:23:58.725 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:23:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:23:58.618+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:23:58.725 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:23:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:23:58.619+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:23:58.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:23:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:23:58.435+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:23:58.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:23:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:23:58.435+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:23:58.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:23:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:23:58.435+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:23:58.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:23:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:23:58.436+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:23:59.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:23:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:23:58.723+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:23:59.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:23:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:23:58.723+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:23:59.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:23:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:23:58.723+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:23:59.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:23:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:23:58.724+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:23:59.693 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:23:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:23:59.440+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:23:59.693 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:23:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:23:59.440+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:23:59.693 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:23:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:23:59.440+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:23:59.693 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:23:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:23:59.441+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:23:59.694 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:23:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:23:59.636+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:23:59.694 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:23:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:23:59.636+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:23:59.694 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:23:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:23:59.636+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:23:59.694 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:23:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:23:59.637+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:24:00.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:23:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:23:59.691+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:24:00.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:23:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:23:59.691+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:24:00.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:23:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:23:59.691+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:24:00.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:23:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:23:59.692+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:24:00.479 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:24:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:24:00.153+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 702 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:24:00.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:24:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:24:00.599+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:24:00.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:24:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:24:00.599+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:24:00.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:24:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:24:00.599+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:24:00.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:24:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:24:00.600+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:24:00.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:24:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:24:00.724+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:24:00.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:24:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:24:00.724+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:24:00.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:24:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:24:00.724+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:24:00.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:24:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:24:00.726+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:24:00.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:24:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:24:00.477+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:24:00.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:24:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:24:00.477+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:24:00.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:24:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:24:00.477+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:24:00.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:24:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:24:00.478+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:24:01.586 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:24:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:24:01.474+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:24:01.586 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:24:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:24:01.474+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:24:01.586 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:24:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:24:01.474+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:24:01.586 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:24:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:24:01.474+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:24:02.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:24:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:24:01.690+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:24:02.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:24:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:24:01.690+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:24:02.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:24:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:24:01.690+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:24:02.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:24:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:24:01.691+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:24:02.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:24:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:24:01.584+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:24:02.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:24:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:24:01.584+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:24:02.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:24:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:24:01.584+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:24:02.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:24:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:24:01.585+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:24:02.720 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:24:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:24:02.623+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:24:02.721 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:24:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:24:02.623+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:24:02.721 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:24:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:24:02.623+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:24:02.721 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:24:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:24:02.623+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:24:02.721 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:24:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:24:02.457+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:24:02.721 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:24:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:24:02.457+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:24:02.721 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:24:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:24:02.457+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:24:02.721 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:24:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:24:02.458+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:24:03.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:24:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:24:02.720+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:24:03.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:24:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:24:02.720+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:24:03.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:24:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:24:02.720+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:24:03.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:24:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:24:02.720+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:24:03.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:24:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:24:03.685+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:24:03.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:24:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:24:03.685+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:24:03.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:24:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:24:03.685+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:24:03.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:24:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:24:03.685+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:24:03.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:24:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:24:03.641+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:24:03.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:24:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:24:03.641+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:24:03.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:24:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:24:03.641+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:24:03.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:24:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:24:03.641+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:24:03.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:24:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:24:03.481+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:24:03.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:24:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:24:03.481+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:24:03.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:24:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:24:03.481+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:24:03.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:24:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:24:03.481+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:24:04.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:24:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:24:04.512+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:24:04.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:24:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:24:04.512+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:24:04.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:24:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:24:04.512+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:24:04.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:24:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:24:04.513+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:24:04.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:24:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:24:04.714+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:24:04.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:24:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:24:04.714+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:24:04.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:24:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:24:04.714+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:24:04.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:24:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:24:04.714+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:24:04.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:24:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:24:04.644+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:24:04.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:24:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:24:04.644+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:24:04.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:24:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:24:04.644+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:24:04.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:24:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:24:04.645+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:24:05.538 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:24:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:24:05.155+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 731 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:24:05.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:24:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:24:05.627+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:24:05.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:24:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:24:05.627+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:24:05.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:24:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:24:05.627+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:24:05.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:24:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:24:05.628+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:24:05.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:24:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:24:05.700+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:24:05.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:24:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:24:05.700+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:24:05.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:24:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:24:05.700+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:24:05.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:24:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:24:05.701+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:24:05.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:24:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:24:05.538+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:24:05.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:24:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:24:05.538+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:24:05.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:24:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:24:05.538+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:24:05.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:24:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:24:05.538+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:24:06.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:24:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:24:06.652+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:24:06.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:24:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:24:06.652+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:24:06.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:24:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:24:06.652+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:24:06.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:24:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:24:06.653+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:24:06.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:24:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:24:06.717+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:24:06.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:24:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:24:06.717+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:24:06.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:24:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:24:06.717+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:24:06.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:24:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:24:06.717+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:24:06.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:24:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:24:06.562+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:24:06.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:24:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:24:06.562+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:24:06.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:24:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:24:06.562+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:24:06.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:24:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:24:06.563+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:24:08.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:24:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:24:07.738+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:24:08.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:24:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:24:07.738+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:24:08.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:24:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:24:07.738+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:24:08.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:24:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:24:07.739+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:24:08.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:24:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:24:07.691+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:24:08.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:24:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:24:07.691+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:24:08.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:24:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:24:07.691+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:24:08.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:24:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:24:07.692+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:24:08.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:24:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:24:07.596+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:24:08.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:24:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:24:07.596+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:24:08.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:24:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:24:07.596+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:24:08.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:24:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:24:07.597+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:24:09.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:24:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:24:08.649+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:24:09.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:24:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:24:08.649+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:24:09.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:24:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:24:08.649+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:24:09.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:24:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:24:08.649+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:24:09.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:24:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:24:08.738+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:24:09.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:24:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:24:08.738+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:24:09.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:24:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:24:08.738+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:24:09.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:24:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:24:08.739+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:24:09.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:24:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:24:08.621+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:24:09.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:24:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:24:08.621+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:24:09.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:24:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:24:08.621+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:24:09.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:24:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:24:08.622+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:24:10.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:24:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:24:09.758+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:24:10.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:24:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:24:09.758+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:24:10.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:24:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:24:09.758+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:24:10.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:24:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:24:09.758+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:24:10.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:24:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:24:09.599+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:24:10.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:24:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:24:09.599+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:24:10.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:24:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:24:09.599+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:24:10.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:24:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:24:09.599+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:24:10.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:24:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:24:09.681+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:24:10.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:24:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:24:09.681+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:24:10.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:24:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:24:09.681+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:24:10.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:24:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:24:09.681+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:24:10.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:24:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:24:10.155+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 758 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:24:11.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:24:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:24:10.772+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:24:11.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:24:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:24:10.772+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:24:11.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:24:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:24:10.772+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:24:11.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:24:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:24:10.773+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:24:11.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:24:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:24:10.613+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:24:11.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:24:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:24:10.613+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:24:11.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:24:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:24:10.613+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:24:11.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:24:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:24:10.613+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:24:11.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:24:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:24:10.645+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:24:11.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:24:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:24:10.645+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:24:11.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:24:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:24:10.645+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:24:11.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:24:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:24:10.646+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:24:12.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:24:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:24:11.791+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:24:12.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:24:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:24:11.791+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:24:12.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:24:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:24:11.791+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:24:12.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:24:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:24:11.792+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:24:12.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:24:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:24:11.592+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:24:12.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:24:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:24:11.592+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:24:12.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:24:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:24:11.592+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:24:12.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:24:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:24:11.592+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:24:12.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:24:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:24:11.632+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:24:12.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:24:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:24:11.632+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:24:12.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:24:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:24:11.632+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:24:12.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:24:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:24:11.633+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:24:12.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:24:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:24:12.549+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:24:12.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:24:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:24:12.549+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:24:12.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:24:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:24:12.549+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:24:12.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:24:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:24:12.549+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:24:12.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:24:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:24:12.590+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:24:12.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:24:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:24:12.590+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:24:12.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:24:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:24:12.590+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:24:12.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:24:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:24:12.591+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:24:13.210 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:24:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:24:12.837+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:24:13.211 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:24:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:24:12.837+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:24:13.211 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:24:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:24:12.837+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:24:13.211 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:24:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:24:12.838+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:24:13.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:24:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:24:13.574+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:24:13.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:24:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:24:13.574+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:24:13.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:24:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:24:13.574+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:24:13.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:24:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:24:13.575+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:24:13.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:24:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:24:13.588+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:24:13.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:24:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:24:13.588+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:24:13.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:24:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:24:13.588+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:24:13.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:24:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:24:13.588+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:24:14.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:24:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:24:13.888+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:24:14.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:24:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:24:13.888+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:24:14.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:24:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:24:13.888+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:24:14.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:24:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:24:13.889+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:24:14.594 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:24:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:24:14.541+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:24:14.595 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:24:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:24:14.541+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:24:14.595 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:24:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:24:14.541+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:24:14.595 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:24:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:24:14.541+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:24:14.886 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:24:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:24:14.594+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:24:14.886 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:24:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:24:14.594+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:24:14.886 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:24:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:24:14.594+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:24:14.886 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:24:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:24:14.594+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:24:15.156 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:24:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:24:14.885+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:24:15.157 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:24:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:24:14.885+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:24:15.157 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:24:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:24:14.885+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:24:15.157 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:24:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:24:14.886+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:24:15.567 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:24:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:24:15.156+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 801 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:24:15.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:24:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:24:15.567+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:24:15.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:24:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:24:15.567+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:24:15.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:24:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:24:15.567+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:24:15.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:24:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:24:15.567+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:24:15.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:24:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:24:15.585+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:24:15.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:24:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:24:15.585+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:24:15.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:24:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:24:15.585+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:24:15.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:24:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:24:15.586+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:24:16.282 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:24:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:24:15.845+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:24:16.283 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:24:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:24:15.845+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:24:16.283 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:24:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:24:15.845+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:24:16.283 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:24:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:24:15.846+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:24:16.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:24:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:24:16.600+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:24:16.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:24:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:24:16.600+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:24:16.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:24:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:24:16.600+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:24:16.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:24:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:24:16.601+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:24:16.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:24:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:24:16.555+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:24:16.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:24:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:24:16.555+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:24:16.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:24:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:24:16.555+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:24:16.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:24:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:24:16.556+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:24:17.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:24:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:24:16.827+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:24:17.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:24:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:24:16.827+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:24:17.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:24:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:24:16.827+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:24:17.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:24:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:24:16.827+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:24:17.812 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:24:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:24:17.556+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:24:17.812 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:24:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:24:17.556+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:24:17.813 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:24:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:24:17.556+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:24:17.813 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:24:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:24:17.557+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:24:17.813 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:24:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:24:17.537+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:24:17.813 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:24:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:24:17.537+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:24:17.813 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:24:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:24:17.537+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:24:17.813 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:24:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:24:17.538+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:24:18.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:24:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:24:17.811+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:24:18.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:24:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:24:17.811+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:24:18.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:24:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:24:17.811+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:24:18.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:24:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:24:17.813+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:24:18.838 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:24:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:24:18.604+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:24:18.838 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:24:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:24:18.604+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:24:18.838 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:24:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:24:18.604+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:24:18.838 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:24:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:24:18.605+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:24:18.839 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:24:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:24:18.585+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:24:18.839 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:24:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:24:18.585+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:24:18.839 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:24:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:24:18.585+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:24:18.839 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:24:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:24:18.585+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:24:19.115 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:24:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:24:18.837+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:24:19.115 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:24:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:24:18.837+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:24:19.115 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:24:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:24:18.837+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:24:19.115 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:24:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:24:18.838+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:24:20.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:24:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:24:19.598+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:24:20.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:24:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:24:19.598+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:24:20.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:24:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:24:19.598+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:24:20.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:24:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:24:19.599+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:24:20.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:24:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:24:19.600+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:24:20.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:24:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:24:19.600+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:24:20.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:24:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:24:19.600+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:24:20.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:24:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:24:19.600+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:24:20.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:24:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:24:19.841+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:24:20.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:24:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:24:19.841+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:24:20.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:24:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:24:19.841+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:24:20.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:24:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:24:19.842+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:24:20.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:24:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:24:20.156+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 827 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:24:20.837 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:24:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:24:20.570+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:24:20.838 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:24:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:24:20.571+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:24:20.838 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:24:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:24:20.571+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:24:20.838 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:24:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:24:20.571+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:24:20.838 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:24:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:24:20.633+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:24:20.838 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:24:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:24:20.633+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:24:20.838 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:24:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:24:20.633+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:24:20.838 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:24:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:24:20.633+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:24:21.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:24:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:24:20.837+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:24:21.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:24:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:24:20.837+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:24:21.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:24:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:24:20.837+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:24:21.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:24:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:24:20.838+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:24:21.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:24:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:24:21.610+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:24:21.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:24:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:24:21.610+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:24:21.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:24:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:24:21.610+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:24:21.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:24:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:24:21.610+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:24:21.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:24:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:24:21.524+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:24:21.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:24:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:24:21.524+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:24:21.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:24:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:24:21.524+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:24:21.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:24:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:24:21.524+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:24:22.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:24:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:24:21.853+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:24:22.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:24:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:24:21.853+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:24:22.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:24:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:24:21.853+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:24:22.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:24:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:24:21.853+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:24:22.806 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:24:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:24:22.538+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:24:22.807 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:24:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:24:22.538+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:24:22.807 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:24:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:24:22.538+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:24:22.807 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:24:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:24:22.538+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:24:22.807 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:24:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:24:22.647+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:24:22.807 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:24:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:24:22.647+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:24:22.807 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:24:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:24:22.647+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:24:22.807 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:24:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:24:22.648+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:24:23.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:24:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:24:22.806+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:24:23.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:24:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:24:22.806+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:24:23.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:24:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:24:22.806+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:24:23.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:24:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:24:22.807+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:24:23.813 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:24:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:24:23.654+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:24:23.813 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:24:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:24:23.654+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:24:23.813 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:24:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:24:23.654+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:24:23.813 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:24:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:24:23.655+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:24:23.814 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:24:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:24:23.540+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:24:23.814 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:24:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:24:23.540+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:24:23.814 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:24:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:24:23.540+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:24:23.814 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:24:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:24:23.541+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:24:24.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:24:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:24:23.812+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:24:24.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:24:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:24:23.812+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:24:24.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:24:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:24:23.812+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:24:24.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:24:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:24:23.813+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:24:24.817 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:24:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:24:24.673+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:24:24.818 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:24:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:24:24.673+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:24:24.818 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:24:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:24:24.673+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:24:24.818 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:24:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:24:24.674+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:24:24.818 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:24:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:24:24.499+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:24:24.818 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:24:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:24:24.499+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:24:24.818 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:24:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:24:24.499+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:24:24.818 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:24:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:24:24.500+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:24:25.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:24:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:24:24.817+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:24:25.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:24:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:24:24.817+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:24:25.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:24:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:24:24.817+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:24:25.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:24:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:24:24.818+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:24:25.455 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:24:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:24:25.157+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 856 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:24:25.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:24:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:24:25.455+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:24:25.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:24:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:24:25.455+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:24:25.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:24:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:24:25.455+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:24:25.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:24:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:24:25.456+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:24:25.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:24:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:24:25.631+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:24:25.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:24:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:24:25.631+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:24:25.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:24:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:24:25.631+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:24:25.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:24:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:24:25.631+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:24:26.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:24:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:24:25.829+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:24:26.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:24:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:24:25.829+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:24:26.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:24:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:24:25.829+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:24:26.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:24:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:24:25.830+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:24:26.798 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:24:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:24:26.649+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:24:26.798 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:24:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:24:26.649+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:24:26.798 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:24:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:24:26.649+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:24:26.798 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:24:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:24:26.650+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:24:26.798 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:24:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:24:26.468+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:24:26.798 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:24:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:24:26.468+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:24:26.799 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:24:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:24:26.468+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:24:26.799 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:24:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:24:26.469+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:24:27.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:24:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:24:26.797+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:24:27.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:24:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:24:26.797+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:24:27.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:24:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:24:26.797+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:24:27.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:24:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:24:26.798+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:24:27.794 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:24:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:24:27.455+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:24:27.794 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:24:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:24:27.455+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:24:27.795 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:24:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:24:27.455+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:24:27.795 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:24:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:24:27.456+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:24:27.795 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:24:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:24:27.653+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:24:27.795 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:24:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:24:27.653+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:24:27.795 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:24:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:24:27.653+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:24:27.795 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:24:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:24:27.654+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:24:28.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:24:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:24:27.793+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:24:28.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:24:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:24:27.793+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:24:28.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:24:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:24:27.793+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:24:28.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:24:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:24:27.795+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:24:28.754 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:24:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:24:28.619+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:24:28.754 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:24:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:24:28.619+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:24:28.754 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:24:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:24:28.619+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:24:28.754 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:24:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:24:28.619+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:24:28.754 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:24:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:24:28.454+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:24:28.754 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:24:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:24:28.454+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:24:28.755 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:24:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:24:28.454+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:24:28.755 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:24:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:24:28.455+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:24:29.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:24:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:24:28.753+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:24:29.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:24:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:24:28.753+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:24:29.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:24:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:24:28.753+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:24:29.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:24:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:24:28.754+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:24:29.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:24:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:24:29.478+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:24:29.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:24:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:24:29.478+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:24:29.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:24:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:24:29.478+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:24:29.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:24:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:24:29.479+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:24:29.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:24:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:24:29.607+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:24:29.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:24:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:24:29.607+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:24:29.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:24:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:24:29.607+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:24:29.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:24:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:24:29.607+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:24:29.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:24:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:24:29.709+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:24:29.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:24:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:24:29.709+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:24:29.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:24:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:24:29.709+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:24:29.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:24:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:24:29.710+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:24:30.478 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:24:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:24:30.157+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 883 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:24:30.749 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:24:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:24:30.477+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:24:30.749 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:24:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:24:30.477+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:24:30.749 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:24:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:24:30.477+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:24:30.749 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:24:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:24:30.478+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:24:30.750 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:24:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:24:30.648+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:24:30.750 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:24:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:24:30.648+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:24:30.750 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:24:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:24:30.648+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:24:30.750 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:24:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:24:30.648+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:24:31.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:24:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:24:30.748+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:24:31.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:24:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:24:30.748+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:24:31.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:24:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:24:30.748+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:24:31.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:24:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:24:30.749+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:24:31.757 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:24:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:24:31.479+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:24:31.757 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:24:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:24:31.479+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:24:31.757 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:24:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:24:31.479+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:24:31.757 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:24:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:24:31.479+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:24:31.757 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:24:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:24:31.678+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:24:31.757 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:24:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:24:31.678+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:24:31.757 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:24:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:24:31.678+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:24:31.757 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:24:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:24:31.678+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:24:32.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:24:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:24:31.756+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:24:32.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:24:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:24:31.756+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:24:32.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:24:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:24:31.756+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:24:32.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:24:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:24:31.757+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:24:32.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:24:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:24:32.468+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:24:32.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:24:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:24:32.468+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:24:32.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:24:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:24:32.468+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:24:32.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:24:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:24:32.469+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:24:32.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:24:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:24:32.660+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:24:32.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:24:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:24:32.660+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:24:32.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:24:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:24:32.660+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:24:32.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:24:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:24:32.661+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:24:33.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:24:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:24:32.729+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:24:33.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:24:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:24:32.729+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:24:33.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:24:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:24:32.729+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:24:33.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:24:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:24:32.730+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:24:33.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:24:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:24:33.740+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:24:33.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:24:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:24:33.740+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:24:33.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:24:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:24:33.740+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:24:33.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:24:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:24:33.740+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:24:33.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:24:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:24:33.692+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:24:33.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:24:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:24:33.692+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:24:33.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:24:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:24:33.692+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:24:33.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:24:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:24:33.693+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:24:33.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:24:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:24:33.509+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:24:33.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:24:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:24:33.509+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:24:33.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:24:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:24:33.509+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:24:33.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:24:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:24:33.510+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:24:34.778 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:24:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:24:34.524+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:24:34.778 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:24:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:24:34.524+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:24:34.779 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:24:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:24:34.524+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:24:34.779 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:24:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:24:34.525+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:24:34.779 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:24:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:24:34.659+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:24:34.779 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:24:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:24:34.659+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:24:34.779 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:24:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:24:34.659+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:24:34.779 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:24:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:24:34.660+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:24:35.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:24:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:24:34.777+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:24:35.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:24:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:24:34.777+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:24:35.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:24:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:24:34.777+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:24:35.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:24:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:24:34.779+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:24:35.565 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:24:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:24:35.158+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 911 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:24:35.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:24:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:24:35.693+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:24:35.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:24:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:24:35.693+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:24:35.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:24:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:24:35.693+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:24:35.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:24:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:24:35.693+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:24:35.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:24:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:24:35.810+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:24:35.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:24:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:24:35.811+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:24:35.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:24:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:24:35.811+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:24:35.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:24:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:24:35.811+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:24:35.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:24:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:24:35.565+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:24:35.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:24:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:24:35.565+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:24:35.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:24:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:24:35.565+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:24:35.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:24:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:24:35.565+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:24:36.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:24:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:24:36.549+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:24:36.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:24:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:24:36.549+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:24:36.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:24:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:24:36.549+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:24:36.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:24:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:24:36.549+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:24:36.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:24:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:24:36.788+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:24:36.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:24:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:24:36.788+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:24:36.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:24:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:24:36.788+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:24:36.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:24:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:24:36.788+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:24:36.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:24:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:24:36.722+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:24:36.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:24:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:24:36.722+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:24:36.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:24:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:24:36.722+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:24:36.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:24:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:24:36.722+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:24:38.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:24:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:24:37.718+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:24:38.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:24:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:24:37.718+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:24:38.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:24:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:24:37.719+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:24:38.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:24:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:24:37.719+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:24:38.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:24:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:24:37.815+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:24:38.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:24:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:24:37.815+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:24:38.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:24:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:24:37.815+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:24:38.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:24:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:24:37.815+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:24:38.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:24:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:24:37.592+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:24:38.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:24:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:24:37.592+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:24:38.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:24:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:24:37.592+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:24:38.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:24:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:24:37.592+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:24:38.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:24:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:24:38.681+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:24:38.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:24:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:24:38.681+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:24:38.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:24:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:24:38.681+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:24:38.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:24:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:24:38.682+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:24:38.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:24:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:24:38.566+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:24:38.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:24:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:24:38.566+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:24:38.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:24:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:24:38.566+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:24:38.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:24:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:24:38.567+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:24:39.115 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:24:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:24:38.842+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:24:39.116 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:24:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:24:38.842+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:24:39.116 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:24:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:24:38.842+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:24:39.116 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:24:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:24:38.842+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:24:40.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:24:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:24:39.654+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:24:40.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:24:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:24:39.654+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:24:40.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:24:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:24:39.654+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:24:40.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:24:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:24:39.655+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:24:40.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:24:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:24:39.831+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:24:40.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:24:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:24:39.831+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:24:40.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:24:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:24:39.831+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:24:40.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:24:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:24:39.831+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:24:40.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:24:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:24:39.608+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:24:40.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:24:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:24:39.608+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:24:40.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:24:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:24:39.608+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:24:40.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:24:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:24:39.609+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:24:40.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:24:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:24:40.159+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 937 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:24:41.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:24:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:24:40.624+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:24:41.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:24:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:24:40.624+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:24:41.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:24:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:24:40.624+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:24:41.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:24:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:24:40.625+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:24:41.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:24:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:24:40.839+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:24:41.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:24:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:24:40.840+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:24:41.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:24:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:24:40.840+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:24:41.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:24:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:24:40.841+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:24:41.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:24:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:24:40.624+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:24:41.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:24:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:24:40.624+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:24:41.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:24:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:24:40.624+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:24:41.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:24:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:24:40.624+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:24:42.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:24:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:24:41.606+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:24:42.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:24:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:24:41.606+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:24:42.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:24:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:24:41.606+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:24:42.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:24:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:24:41.607+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:24:42.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:24:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:24:41.672+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:24:42.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:24:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:24:41.672+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:24:42.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:24:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:24:41.672+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:24:42.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:24:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:24:41.673+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:24:42.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:24:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:24:41.812+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:24:42.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:24:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:24:41.812+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:24:42.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:24:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:24:41.812+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:24:42.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:24:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:24:41.813+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:24:42.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:24:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:24:42.848+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:24:42.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:24:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:24:42.848+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:24:42.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:24:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:24:42.848+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:24:42.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:24:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:24:42.848+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:24:42.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:24:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:24:42.618+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:24:42.971 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:24:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:24:42.618+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:24:42.971 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:24:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:24:42.618+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:24:42.971 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:24:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:24:42.619+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:24:42.971 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:24:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:24:42.637+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:24:42.971 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:24:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:24:42.637+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:24:42.971 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:24:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:24:42.637+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:24:42.971 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:24:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:24:42.638+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:24:43.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:24:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:24:43.652+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:24:43.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:24:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:24:43.652+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:24:43.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:24:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:24:43.652+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:24:43.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:24:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:24:43.653+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:24:43.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:24:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:24:43.596+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:24:43.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:24:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:24:43.596+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:24:43.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:24:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:24:43.596+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:24:43.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:24:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:24:43.596+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:24:44.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:24:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:24:43.824+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:24:44.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:24:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:24:43.824+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:24:44.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:24:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:24:43.824+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:24:44.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:24:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:24:43.825+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:24:45.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:24:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:24:44.808+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:24:45.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:24:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:24:44.808+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:24:45.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:24:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:24:44.808+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:24:45.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:24:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:24:44.809+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:24:45.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:24:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:24:44.589+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:24:45.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:24:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:24:44.589+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:24:45.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:24:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:24:44.589+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:24:45.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:24:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:24:44.589+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:24:45.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:24:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:24:44.636+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:24:45.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:24:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:24:44.636+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:24:45.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:24:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:24:44.636+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:24:45.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:24:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:24:44.637+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:24:45.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:24:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:24:45.159+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 966 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:24:46.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:24:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:24:45.827+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:24:46.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:24:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:24:45.827+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:24:46.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:24:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:24:45.827+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:24:46.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:24:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:24:45.828+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:24:46.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:24:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:24:45.597+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:24:46.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:24:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:24:45.597+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:24:46.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:24:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:24:45.597+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:24:46.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:24:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:24:45.598+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:24:46.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:24:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:24:45.598+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:24:46.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:24:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:24:45.598+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:24:46.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:24:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:24:45.598+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:24:46.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:24:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:24:45.599+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:24:46.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:24:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:24:46.559+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:24:46.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:24:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:24:46.559+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:24:46.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:24:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:24:46.559+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:24:46.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:24:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:24:46.559+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:24:46.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:24:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:24:46.611+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:24:46.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:24:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:24:46.611+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:24:46.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:24:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:24:46.611+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:24:46.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:24:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:24:46.612+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:24:47.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:24:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:24:46.834+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:24:47.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:24:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:24:46.834+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:24:47.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:24:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:24:46.834+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:24:47.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:24:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:24:46.835+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:24:47.853 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:24:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:24:47.608+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:24:47.854 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:24:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:24:47.608+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:24:47.854 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:24:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:24:47.608+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:24:47.854 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:24:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:24:47.608+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:24:47.854 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:24:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:24:47.596+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:24:47.854 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:24:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:24:47.596+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:24:47.854 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:24:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:24:47.596+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:24:47.854 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:24:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:24:47.596+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:24:48.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:24:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:24:47.853+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:24:48.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:24:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:24:47.853+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:24:48.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:24:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:24:47.853+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:24:48.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:24:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:24:47.853+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:24:48.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:24:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:24:48.604+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:24:48.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:24:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:24:48.604+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:24:48.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:24:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:24:48.604+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:24:48.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:24:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:24:48.605+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:24:48.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:24:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:24:48.562+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:24:48.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:24:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:24:48.562+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:24:48.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:24:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:24:48.562+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:24:48.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:24:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:24:48.562+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:24:49.116 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:24:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:24:48.860+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:24:49.116 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:24:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:24:48.860+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:24:49.116 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:24:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:24:48.860+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:24:49.116 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:24:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:24:48.862+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:24:49.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:24:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:24:49.635+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:24:49.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:24:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:24:49.635+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:24:49.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:24:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:24:49.635+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:24:49.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:24:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:24:49.636+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:24:49.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:24:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:24:49.541+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:24:49.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:24:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:24:49.541+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:24:49.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:24:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:24:49.541+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:24:49.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:24:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:24:49.541+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:24:50.161 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:24:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:24:49.904+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:24:50.161 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:24:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:24:49.904+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:24:50.161 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:24:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:24:49.904+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:24:50.161 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:24:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:24:49.906+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:24:50.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:24:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:24:50.160+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 996 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:24:50.915 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:24:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:24:50.676+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:24:50.916 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:24:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:24:50.676+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:24:50.916 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:24:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:24:50.676+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:24:50.916 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:24:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:24:50.676+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:24:50.916 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:24:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:24:50.572+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:24:50.916 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:24:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:24:50.572+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:24:50.916 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:24:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:24:50.572+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:24:50.916 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:24:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:24:50.572+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:24:51.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:24:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:24:50.915+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:24:51.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:24:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:24:50.915+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:24:51.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:24:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:24:50.915+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:24:51.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:24:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:24:50.915+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:24:51.958 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:24:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:24:51.655+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:24:51.958 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:24:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:24:51.655+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:24:51.958 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:24:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:24:51.655+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:24:51.958 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:24:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:24:51.655+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:24:51.959 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:24:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:24:51.614+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:24:51.959 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:24:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:24:51.614+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:24:51.959 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:24:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:24:51.614+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:24:51.959 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:24:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:24:51.615+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:24:52.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:24:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:24:51.957+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:24:52.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:24:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:24:51.957+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:24:52.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:24:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:24:51.957+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:24:52.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:24:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:24:51.958+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:24:52.916 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:24:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:24:52.573+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:24:52.916 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:24:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:24:52.573+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:24:52.917 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:24:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:24:52.573+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:24:52.917 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:24:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:24:52.574+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:24:52.917 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:24:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:24:52.612+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:24:52.917 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:24:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:24:52.612+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:24:52.917 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:24:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:24:52.612+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:24:52.917 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:24:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:24:52.613+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:24:53.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:24:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:24:52.916+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:24:53.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:24:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:24:52.916+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:24:53.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:24:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:24:52.916+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:24:53.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:24:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:24:52.916+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:24:53.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:24:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:24:53.567+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:24:53.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:24:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:24:53.567+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:24:53.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:24:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:24:53.567+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:24:53.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:24:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:24:53.567+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:24:53.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:24:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:24:53.523+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:24:53.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:24:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:24:53.523+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:24:53.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:24:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:24:53.523+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:24:53.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:24:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:24:53.523+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:24:54.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:24:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:24:53.933+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:24:54.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:24:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:24:53.933+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:24:54.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:24:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:24:53.933+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:24:54.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:24:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:24:53.933+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:24:54.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:24:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:24:54.489+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:24:54.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:24:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:24:54.489+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:24:54.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:24:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:24:54.489+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:24:54.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:24:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:24:54.490+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:24:54.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:24:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:24:54.551+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:24:54.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:24:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:24:54.551+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:24:54.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:24:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:24:54.551+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:24:54.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:24:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:24:54.551+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:24:55.285 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:24:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:24:55.159+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 1030 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:24:55.285 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:24:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:24:54.973+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:24:55.285 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:24:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:24:54.973+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:24:55.285 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:24:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:24:54.973+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:24:55.285 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:24:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:24:54.974+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:24:55.555 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:24:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:24:55.481+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:24:55.555 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:24:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:24:55.481+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:24:55.556 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:24:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:24:55.481+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:24:55.556 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:24:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:24:55.481+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:24:55.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:24:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:24:55.554+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:24:55.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:24:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:24:55.554+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:24:55.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:24:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:24:55.554+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:24:55.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:24:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:24:55.555+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:24:56.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:24:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:24:56.008+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:24:56.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:24:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:24:56.008+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:24:56.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:24:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:24:56.008+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:24:56.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:24:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:24:56.009+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:24:56.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:24:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:24:56.459+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:24:56.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:24:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:24:56.459+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:24:56.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:24:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:24:56.459+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:24:56.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:24:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:24:56.459+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:24:56.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:24:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:24:56.556+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:24:56.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:24:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:24:56.556+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:24:56.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:24:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:24:56.556+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:24:56.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:24:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:24:56.557+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:24:57.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:24:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:24:56.959+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:24:57.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:24:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:24:56.959+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:24:57.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:24:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:24:56.959+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:24:57.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:24:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:24:56.960+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:24:57.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:24:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:24:57.489+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:24:57.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:24:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:24:57.489+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:24:57.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:24:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:24:57.489+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:24:57.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:24:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:24:57.489+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:24:57.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:24:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:24:57.601+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:24:57.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:24:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:24:57.601+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:24:57.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:24:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:24:57.601+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:24:57.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:24:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:24:57.602+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:24:58.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:24:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:24:57.959+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:24:58.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:24:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:24:57.959+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:24:58.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:24:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:24:57.959+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:24:58.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:24:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:24:57.960+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:24:58.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:24:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:24:58.557+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:24:58.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:24:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:24:58.557+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:24:58.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:24:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:24:58.557+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:24:58.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:24:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:24:58.557+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:24:58.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:24:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:24:58.476+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:24:58.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:24:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:24:58.476+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:24:58.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:24:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:24:58.476+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:24:58.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:24:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:24:58.477+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:24:59.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:24:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:24:58.941+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:24:59.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:24:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:24:58.941+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:24:59.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:24:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:24:58.941+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:24:59.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:24:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:24:58.941+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:24:59.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:24:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:24:59.575+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:24:59.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:24:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:24:59.575+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:24:59.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:24:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:24:59.575+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:24:59.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:24:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:24:59.576+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:24:59.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:24:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:24:59.441+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:24:59.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:24:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:24:59.441+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:24:59.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:24:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:24:59.441+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:24:59.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:24:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:24:59.442+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:25:00.185 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:25:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:25:00.160+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 1072 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:25:00.185 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:24:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:24:59.923+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:25:00.185 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:24:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:24:59.923+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:25:00.185 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:24:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:24:59.923+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:25:00.185 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:24:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:24:59.923+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:25:00.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:25:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:25:00.474+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:25:00.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:25:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:25:00.474+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:25:00.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:25:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:25:00.474+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:25:00.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:25:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:25:00.474+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:25:00.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:25:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:25:00.562+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:25:00.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:25:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:25:00.562+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:25:00.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:25:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:25:00.562+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:25:00.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:25:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:25:00.563+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:25:01.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:25:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:25:00.959+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:25:01.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:25:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:25:00.959+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:25:01.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:25:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:25:00.959+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:25:01.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:25:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:25:00.959+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:25:01.496 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:25:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:25:01.493+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:25:01.496 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:25:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:25:01.493+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:25:01.496 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:25:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:25:01.493+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:25:01.496 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:25:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:25:01.493+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:25:01.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:25:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:25:01.559+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:25:01.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:25:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:25:01.559+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:25:01.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:25:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:25:01.559+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:25:01.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:25:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:25:01.559+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:25:02.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:25:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:25:01.971+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:25:02.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:25:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:25:01.971+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:25:02.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:25:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:25:01.971+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:25:02.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:25:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:25:01.971+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:25:02.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:25:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:25:02.536+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:25:02.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:25:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:25:02.536+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:25:02.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:25:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:25:02.536+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:25:02.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:25:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:25:02.536+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:25:02.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:25:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:25:02.529+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:25:02.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:25:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:25:02.529+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:25:02.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:25:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:25:02.529+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:25:02.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:25:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:25:02.529+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:25:03.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:25:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:25:02.997+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:25:03.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:25:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:25:02.997+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:25:03.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:25:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:25:02.997+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:25:03.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:25:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:25:02.998+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:25:03.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:25:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:25:03.542+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:25:03.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:25:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:25:03.542+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:25:03.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:25:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:25:03.542+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:25:03.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:25:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:25:03.543+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:25:03.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:25:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:25:03.549+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:25:03.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:25:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:25:03.549+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:25:03.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:25:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:25:03.549+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:25:03.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:25:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:25:03.550+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:25:04.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:25:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:25:04.014+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:25:04.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:25:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:25:04.014+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:25:04.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:25:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:25:04.014+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:25:04.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:25:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:25:04.014+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:25:04.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:25:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:25:04.504+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:25:04.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:25:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:25:04.504+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:25:04.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:25:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:25:04.504+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:25:04.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:25:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:25:04.504+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:25:04.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:25:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:25:04.539+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:25:04.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:25:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:25:04.539+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:25:04.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:25:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:25:04.539+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:25:04.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:25:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:25:04.540+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:25:05.283 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:25:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:25:05.162+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 1117 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:25:05.283 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:25:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:25:05.013+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:25:05.283 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:25:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:25:05.013+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:25:05.283 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:25:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:25:05.013+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:25:05.283 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:25:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:25:05.014+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:25:05.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:25:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:25:05.509+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:25:05.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:25:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:25:05.509+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:25:05.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:25:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:25:05.509+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:25:05.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:25:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:25:05.510+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:25:05.569 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:25:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:25:05.521+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:25:05.569 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:25:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:25:05.521+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:25:05.569 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:25:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:25:05.521+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:25:05.569 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:25:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:25:05.521+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:25:06.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:25:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:25:06.005+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:25:06.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:25:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:25:06.005+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:25:06.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:25:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:25:06.005+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:25:06.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:25:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:25:06.006+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:25:06.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:25:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:25:06.518+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:25:06.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:25:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:25:06.518+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:25:06.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:25:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:25:06.518+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:25:06.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:25:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:25:06.519+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:25:06.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:25:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:25:06.553+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:25:06.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:25:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:25:06.553+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:25:06.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:25:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:25:06.553+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:25:06.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:25:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:25:06.553+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:25:07.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:25:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:25:07.040+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:25:07.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:25:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:25:07.040+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:25:07.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:25:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:25:07.040+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:25:07.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:25:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:25:07.041+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:25:07.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:25:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:25:07.551+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:25:07.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:25:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:25:07.551+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:25:07.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:25:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:25:07.551+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:25:07.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:25:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:25:07.551+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:25:07.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:25:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:25:07.518+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:25:07.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:25:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:25:07.518+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:25:07.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:25:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:25:07.518+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:25:07.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:25:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:25:07.519+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:25:08.508 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:25:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:25:08.085+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:25:08.509 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:25:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:25:08.085+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:25:08.509 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:25:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:25:08.085+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:25:08.509 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:25:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:25:08.085+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:25:08.574 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:25:08.573+0000 7ff67b6c3700 0 monclient(hunting): authenticate timed out after 300 2024-03-18T01:25:08.575 INFO:teuthology.orchestra.run.smithi029.stderr:[errno 110] RADOS timed out (error connecting to the cluster) 2024-03-18T01:25:08.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:25:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:25:08.574+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:25:08.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:25:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:25:08.574+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:25:08.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:25:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:25:08.574+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:25:08.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:25:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:25:08.575+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:25:08.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:25:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:25:08.513+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:25:08.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:25:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:25:08.513+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:25:08.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:25:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:25:08.513+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:25:08.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:25:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:25:08.513+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:25:09.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:25:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:25:09.127+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:25:09.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:25:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:25:09.127+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:25:09.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:25:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:25:09.127+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:25:09.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:25:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:25:09.128+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:25:09.345 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 37f51182-e4c4-11ee-95c9-87774f69a715 -e sha1=3d78f5ba07bf18880172c07d4d7b4ac00c9d30af -- bash -c 'ceph orch ps' 2024-03-18T01:25:09.606 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:25:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:25:09.531+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:25:09.606 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:25:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:25:09.531+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:25:09.607 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:25:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:25:09.531+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:25:09.607 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:25:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:25:09.532+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:25:09.615 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/37f51182-e4c4-11ee-95c9-87774f69a715/mon.smithi029/config 2024-03-18T01:25:09.892 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:25:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:25:09.605+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:25:09.893 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:25:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:25:09.605+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:25:09.893 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:25:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:25:09.605+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:25:09.893 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:25:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:25:09.606+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:25:10.163 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:25:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:25:10.100+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:25:10.163 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:25:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:25:10.100+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:25:10.164 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:25:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:25:10.100+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:25:10.164 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:25:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:25:10.101+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:25:10.545 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:25:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:25:10.162+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 1160 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:25:10.804 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:25:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:25:10.641+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:25:10.804 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:25:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:25:10.641+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:25:10.804 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:25:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:25:10.641+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:25:10.805 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:25:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:25:10.642+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:25:10.805 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:25:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:25:10.544+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:25:10.805 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:25:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:25:10.545+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:25:10.805 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:25:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:25:10.545+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:25:10.805 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:25:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:25:10.545+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:25:11.364 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:25:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:25:11.087+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:25:11.364 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:25:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:25:11.087+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:25:11.364 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:25:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:25:11.087+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:25:11.364 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:25:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:25:11.087+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:25:11.641 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:25:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:25:11.607+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:25:11.642 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:25:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:25:11.607+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:25:11.642 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:25:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:25:11.607+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:25:11.642 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:25:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:25:11.608+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:25:11.642 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:25:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:25:11.506+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:25:11.642 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:25:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:25:11.506+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:25:11.642 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:25:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:25:11.506+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:25:11.642 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:25:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:25:11.507+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:25:12.557 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:25:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:25:12.134+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:25:12.557 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:25:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:25:12.134+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:25:12.557 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:25:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:25:12.134+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:25:12.557 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:25:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:25:12.135+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:25:12.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:25:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:25:12.556+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:25:12.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:25:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:25:12.556+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:25:12.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:25:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:25:12.556+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:25:12.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:25:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:25:12.557+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:25:12.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:25:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:25:12.652+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:25:12.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:25:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:25:12.652+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:25:12.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:25:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:25:12.652+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:25:12.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:25:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:25:12.653+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:25:13.527 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:25:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:25:13.118+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:25:13.527 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:25:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:25:13.118+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:25:13.528 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:25:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:25:13.118+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:25:13.528 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:25:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:25:13.119+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:25:13.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:25:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:25:13.527+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:25:13.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:25:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:25:13.527+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:25:13.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:25:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:25:13.527+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:25:13.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:25:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:25:13.527+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:25:13.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:25:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:25:13.685+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:25:13.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:25:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:25:13.685+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:25:13.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:25:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:25:13.685+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:25:13.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:25:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:25:13.686+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:25:14.558 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:25:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:25:14.129+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:25:14.558 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:25:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:25:14.129+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:25:14.559 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:25:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:25:14.129+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:25:14.559 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:25:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:25:14.129+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:25:14.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:25:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:25:14.557+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:25:14.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:25:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:25:14.558+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:25:14.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:25:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:25:14.558+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:25:14.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:25:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:25:14.558+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:25:14.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:25:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:25:14.716+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:25:14.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:25:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:25:14.716+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:25:14.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:25:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:25:14.716+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:25:14.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:25:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:25:14.717+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:25:15.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:25:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:25:15.163+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 1204 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:25:15.569 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:25:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:25:15.149+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:25:15.569 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:25:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:25:15.149+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:25:15.570 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:25:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:25:15.149+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:25:15.570 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:25:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:25:15.150+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:25:16.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:25:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:25:15.753+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:25:16.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:25:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:25:15.753+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:25:16.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:25:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:25:15.753+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:25:16.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:25:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:25:15.754+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:25:16.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:25:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:25:15.599+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:25:16.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:25:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:25:15.599+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:25:16.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:25:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:25:15.599+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:25:16.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:25:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:25:15.600+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:25:16.569 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:25:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:25:16.193+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:25:16.569 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:25:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:25:16.193+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:25:16.569 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:25:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:25:16.193+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:25:16.569 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:25:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:25:16.194+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:25:17.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:25:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:25:16.619+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:25:17.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:25:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:25:16.619+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:25:17.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:25:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:25:16.619+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:25:17.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:25:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:25:16.620+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:25:17.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:25:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:25:16.793+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:25:17.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:25:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:25:16.793+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:25:17.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:25:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:25:16.793+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:25:17.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:25:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:25:16.794+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:25:17.569 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:25:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:25:17.175+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:25:17.569 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:25:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:25:17.175+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:25:17.569 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:25:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:25:17.175+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:25:17.569 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:25:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:25:17.175+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:25:18.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:25:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:25:17.756+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:25:18.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:25:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:25:17.756+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:25:18.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:25:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:25:17.756+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:25:18.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:25:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:25:17.757+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:25:18.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:25:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:25:17.621+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:25:18.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:25:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:25:17.621+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:25:18.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:25:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:25:17.621+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:25:18.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:25:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:25:17.621+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:25:18.569 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:25:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:25:18.149+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:25:18.569 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:25:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:25:18.149+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:25:18.569 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:25:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:25:18.149+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:25:18.569 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:25:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:25:18.149+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:25:19.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:25:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:25:18.594+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:25:19.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:25:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:25:18.594+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:25:19.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:25:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:25:18.594+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:25:19.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:25:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:25:18.594+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:25:19.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:25:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:25:18.717+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:25:19.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:25:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:25:18.717+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:25:19.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:25:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:25:18.717+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:25:19.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:25:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:25:18.717+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:25:19.569 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:25:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:25:19.150+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:25:19.569 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:25:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:25:19.150+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:25:19.570 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:25:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:25:19.150+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:25:19.570 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:25:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:25:19.151+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:25:20.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:25:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:25:19.720+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:25:20.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:25:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:25:19.720+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:25:20.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:25:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:25:19.720+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:25:20.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:25:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:25:19.720+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:25:20.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:25:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:25:19.578+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:25:20.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:25:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:25:19.578+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:25:20.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:25:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:25:19.578+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:25:20.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:25:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:25:19.578+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:25:20.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:25:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:25:20.163+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 1258 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:25:20.569 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:25:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:25:20.134+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:25:20.569 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:25:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:25:20.134+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:25:20.569 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:25:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:25:20.134+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:25:20.569 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:25:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:25:20.135+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:25:21.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:25:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:25:20.702+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:25:21.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:25:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:25:20.702+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:25:21.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:25:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:25:20.702+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:25:21.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:25:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:25:20.702+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:25:21.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:25:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:25:20.611+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:25:21.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:25:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:25:20.611+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:25:21.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:25:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:25:20.611+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:25:21.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:25:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:25:20.611+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:25:21.569 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:25:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:25:21.168+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:25:21.569 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:25:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:25:21.168+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:25:21.569 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:25:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:25:21.168+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:25:21.569 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:25:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:25:21.169+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:25:22.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:25:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:25:21.727+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:25:22.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:25:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:25:21.727+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:25:22.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:25:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:25:21.727+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:25:22.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:25:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:25:21.727+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:25:22.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:25:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:25:21.579+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:25:22.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:25:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:25:21.579+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:25:22.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:25:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:25:21.579+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:25:22.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:25:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:25:21.580+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:25:22.570 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:25:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:25:22.212+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:25:22.570 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:25:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:25:22.212+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:25:22.570 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:25:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:25:22.212+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:25:22.570 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:25:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:25:22.213+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:25:23.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:25:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:25:22.697+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:25:23.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:25:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:25:22.697+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:25:23.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:25:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:25:22.697+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:25:23.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:25:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:25:22.697+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:25:23.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:25:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:25:22.620+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:25:23.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:25:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:25:22.620+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:25:23.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:25:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:25:22.620+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:25:23.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:25:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:25:22.621+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:25:23.528 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:25:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:25:23.223+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:25:23.529 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:25:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:25:23.223+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:25:23.529 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:25:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:25:23.223+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:25:23.529 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:25:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:25:23.224+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:25:23.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:25:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:25:23.573+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:25:23.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:25:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:25:23.573+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:25:23.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:25:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:25:23.573+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:25:23.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:25:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:25:23.574+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:25:23.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:25:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:25:23.673+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:25:23.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:25:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:25:23.673+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:25:23.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:25:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:25:23.673+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:25:23.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:25:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:25:23.673+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:25:24.569 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:25:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:25:24.176+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:25:24.569 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:25:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:25:24.176+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:25:24.569 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:25:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:25:24.176+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:25:24.569 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:25:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:25:24.177+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:25:25.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:25:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:25:24.625+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:25:25.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:25:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:25:24.625+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:25:25.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:25:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:25:24.625+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:25:25.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:25:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:25:24.626+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:25:25.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:25:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:25:24.620+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:25:25.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:25:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:25:24.620+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:25:25.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:25:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:25:24.620+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:25:25.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:25:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:25:24.620+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:25:25.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:25:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:25:25.164+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 1352 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:25:25.569 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:25:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:25:25.193+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:25:25.569 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:25:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:25:25.193+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:25:25.569 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:25:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:25:25.193+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:25:25.570 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:25:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:25:25.194+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:25:25.898 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:25:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:25:25.625+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:25:25.898 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:25:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:25:25.625+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:25:25.898 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:25:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:25:25.625+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:25:25.898 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:25:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:25:25.626+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:25:25.899 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:25:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:25:25.616+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:25:25.899 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:25:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:25:25.616+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:25:25.899 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:25:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:25:25.616+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:25:25.899 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:25:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:25:25.616+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:25:26.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:25:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:25:26.144+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:25:26.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:25:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:25:26.144+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:25:26.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:25:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:25:26.144+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:25:26.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:25:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:25:26.145+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:25:26.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:25:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:25:26.609+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:25:26.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:25:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:25:26.609+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:25:26.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:25:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:25:26.610+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:25:26.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:25:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:25:26.610+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:25:26.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:25:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:25:26.567+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:25:26.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:25:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:25:26.567+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:25:26.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:25:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:25:26.567+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:25:26.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:25:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:25:26.568+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:25:27.569 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:25:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:25:27.111+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:25:27.569 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:25:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:25:27.111+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:25:27.569 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:25:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:25:27.111+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:25:27.569 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:25:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:25:27.112+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:25:28.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:25:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:25:27.586+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:25:28.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:25:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:25:27.586+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:25:28.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:25:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:25:27.586+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:25:28.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:25:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:25:27.587+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:25:28.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:25:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:25:27.575+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:25:28.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:25:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:25:27.575+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:25:28.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:25:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:25:27.575+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:25:28.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:25:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:25:27.575+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:25:28.569 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:25:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:25:28.110+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:25:28.569 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:25:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:25:28.110+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:25:28.569 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:25:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:25:28.110+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:25:28.569 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:25:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:25:28.111+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:25:29.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:25:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:25:28.580+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:25:29.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:25:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:25:28.580+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:25:29.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:25:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:25:28.580+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:25:29.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:25:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:25:28.581+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:25:29.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:25:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:25:28.592+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:25:29.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:25:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:25:28.592+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:25:29.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:25:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:25:28.592+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:25:29.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:25:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:25:28.592+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:25:29.536 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:25:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:25:29.075+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:25:29.536 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:25:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:25:29.075+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:25:29.536 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:25:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:25:29.075+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:25:29.536 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:25:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:25:29.076+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:25:29.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:25:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:25:29.535+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:25:29.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:25:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:25:29.535+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:25:29.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:25:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:25:29.535+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:25:29.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:25:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:25:29.536+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:25:29.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:25:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:25:29.586+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:25:29.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:25:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:25:29.586+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:25:29.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:25:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:25:29.586+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:25:29.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:25:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:25:29.587+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:25:30.287 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:25:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:25:30.164+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 1445 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:25:30.287 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:25:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:25:30.119+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:25:30.287 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:25:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:25:30.119+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:25:30.287 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:25:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:25:30.119+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:25:30.287 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:25:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:25:30.120+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:25:30.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:25:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:25:30.551+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:25:30.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:25:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:25:30.551+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:25:30.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:25:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:25:30.551+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:25:30.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:25:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:25:30.552+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:25:30.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:25:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:25:30.550+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:25:30.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:25:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:25:30.550+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:25:30.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:25:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:25:30.550+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:25:30.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:25:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:25:30.551+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:25:31.520 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:25:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:25:31.137+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:25:31.521 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:25:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:25:31.137+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:25:31.521 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:25:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:25:31.137+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:25:31.521 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:25:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:25:31.137+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:25:31.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:25:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:25:31.520+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:25:31.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:25:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:25:31.520+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:25:31.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:25:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:25:31.520+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:25:31.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:25:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:25:31.520+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:25:31.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:25:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:25:31.559+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:25:31.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:25:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:25:31.559+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:25:31.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:25:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:25:31.559+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:25:31.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:25:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:25:31.559+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:25:32.540 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:25:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:25:32.158+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:25:32.541 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:25:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:25:32.158+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:25:32.541 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:25:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:25:32.158+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:25:32.541 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:25:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:25:32.159+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:25:32.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:25:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:25:32.540+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:25:32.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:25:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:25:32.540+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:25:32.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:25:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:25:32.540+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:25:32.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:25:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:25:32.540+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:25:32.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:25:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:25:32.555+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:25:32.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:25:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:25:32.555+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:25:32.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:25:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:25:32.555+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:25:32.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:25:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:25:32.555+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:25:33.529 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:25:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:25:33.114+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:25:33.530 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:25:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:25:33.114+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:25:33.530 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:25:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:25:33.114+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:25:33.530 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:25:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:25:33.115+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:25:33.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:25:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:25:33.539+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:25:33.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:25:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:25:33.539+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:25:33.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:25:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:25:33.539+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:25:33.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:25:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:25:33.540+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:25:33.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:25:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:25:33.529+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:25:33.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:25:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:25:33.529+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:25:33.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:25:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:25:33.529+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:25:33.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:25:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:25:33.529+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:25:34.502 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:25:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:25:34.098+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:25:34.502 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:25:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:25:34.098+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:25:34.502 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:25:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:25:34.098+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:25:34.502 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:25:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:25:34.099+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:25:34.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:25:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:25:34.501+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:25:34.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:25:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:25:34.501+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:25:34.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:25:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:25:34.501+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:25:34.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:25:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:25:34.502+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:25:34.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:25:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:25:34.553+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:25:34.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:25:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:25:34.553+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:25:34.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:25:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:25:34.553+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:25:34.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:25:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:25:34.554+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:25:35.530 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:25:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:25:35.165+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 1539 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:25:35.530 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:25:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:25:35.099+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:25:35.530 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:25:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:25:35.099+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:25:35.530 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:25:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:25:35.099+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:25:35.530 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:25:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:25:35.100+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:25:35.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:25:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:25:35.566+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:25:35.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:25:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:25:35.566+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:25:35.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:25:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:25:35.566+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:25:35.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:25:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:25:35.567+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:25:35.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:25:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:25:35.529+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:25:35.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:25:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:25:35.529+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:25:35.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:25:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:25:35.529+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:25:35.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:25:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:25:35.530+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:25:36.526 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:25:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:25:36.077+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:25:36.526 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:25:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:25:36.077+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:25:36.526 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:25:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:25:36.077+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:25:36.526 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:25:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:25:36.078+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:25:36.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:25:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:25:36.536+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:25:36.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:25:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:25:36.536+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:25:36.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:25:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:25:36.536+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:25:36.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:25:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:25:36.537+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:25:36.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:25:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:25:36.525+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:25:36.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:25:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:25:36.525+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:25:36.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:25:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:25:36.525+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:25:36.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:25:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:25:36.526+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:25:37.494 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:25:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:25:37.070+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:25:37.494 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:25:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:25:37.070+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:25:37.494 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:25:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:25:37.070+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:25:37.494 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:25:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:25:37.071+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:25:37.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:25:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:25:37.586+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:25:37.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:25:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:25:37.586+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:25:37.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:25:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:25:37.586+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:25:37.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:25:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:25:37.587+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:25:37.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:25:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:25:37.493+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:25:37.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:25:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:25:37.493+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:25:37.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:25:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:25:37.493+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:25:37.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:25:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:25:37.494+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:25:38.492 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:25:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:25:38.086+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:25:38.493 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:25:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:25:38.086+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:25:38.493 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:25:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:25:38.086+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:25:38.493 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:25:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:25:38.087+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:25:38.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:25:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:25:38.546+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:25:38.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:25:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:25:38.546+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:25:38.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:25:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:25:38.546+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:25:38.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:25:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:25:38.546+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:25:38.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:25:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:25:38.492+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:25:38.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:25:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:25:38.492+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:25:38.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:25:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:25:38.492+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:25:38.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:25:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:25:38.492+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:25:39.472 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:25:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:25:39.073+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:25:39.473 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:25:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:25:39.073+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:25:39.473 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:25:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:25:39.073+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:25:39.473 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:25:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:25:39.074+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:25:39.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:25:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:25:39.472+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:25:39.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:25:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:25:39.472+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:25:39.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:25:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:25:39.472+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:25:39.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:25:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:25:39.472+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:25:39.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:25:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:25:39.532+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:25:39.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:25:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:25:39.532+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:25:39.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:25:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:25:39.532+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:25:39.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:25:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:25:39.533+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:25:40.469 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:25:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:25:40.166+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 1630 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:25:40.469 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:25:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:25:40.075+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:25:40.469 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:25:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:25:40.075+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:25:40.469 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:25:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:25:40.075+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:25:40.469 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:25:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:25:40.076+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:25:40.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:25:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:25:40.583+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:25:40.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:25:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:25:40.583+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:25:40.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:25:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:25:40.583+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:25:40.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:25:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:25:40.583+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:25:40.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:25:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:25:40.468+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:25:40.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:25:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:25:40.468+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:25:40.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:25:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:25:40.468+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:25:40.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:25:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:25:40.469+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:25:41.500 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:25:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:25:41.076+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:25:41.501 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:25:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:25:41.076+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:25:41.501 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:25:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:25:41.076+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:25:41.501 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:25:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:25:41.076+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:25:41.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:25:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:25:41.499+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:25:41.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:25:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:25:41.499+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:25:41.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:25:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:25:41.499+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:25:41.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:25:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:25:41.500+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:25:41.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:25:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:25:41.617+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:25:41.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:25:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:25:41.617+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:25:41.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:25:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:25:41.617+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:25:41.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:25:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:25:41.618+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:25:42.299 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:25:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:25:42.042+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:25:42.299 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:25:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:25:42.042+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:25:42.300 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:25:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:25:42.042+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:25:42.300 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:25:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:25:42.043+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:25:42.569 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:25:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:25:42.489+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:25:42.569 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:25:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:25:42.489+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:25:42.569 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:25:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:25:42.489+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:25:42.569 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:25:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:25:42.489+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:25:43.047 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:25:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:25:42.621+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:25:43.047 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:25:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:25:42.621+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:25:43.047 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:25:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:25:42.621+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:25:43.047 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:25:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:25:42.622+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:25:43.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:25:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:25:43.046+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:25:43.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:25:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:25:43.046+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:25:43.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:25:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:25:43.046+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:25:43.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:25:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:25:43.047+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:25:43.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:25:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:25:43.648+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:25:43.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:25:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:25:43.648+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:25:43.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:25:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:25:43.648+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:25:43.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:25:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:25:43.649+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:25:43.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:25:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:25:43.470+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:25:43.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:25:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:25:43.470+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:25:43.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:25:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:25:43.470+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:25:43.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:25:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:25:43.470+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:25:44.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:25:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:25:44.032+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:25:44.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:25:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:25:44.032+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:25:44.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:25:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:25:44.032+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:25:44.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:25:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:25:44.033+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:25:44.678 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:25:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:25:44.478+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:25:44.678 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:25:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:25:44.478+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:25:44.678 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:25:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:25:44.478+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:25:44.678 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:25:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:25:44.478+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:25:44.984 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:25:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:25:44.677+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:25:44.985 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:25:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:25:44.677+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:25:44.985 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:25:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:25:44.677+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:25:44.985 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:25:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:25:44.678+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:25:45.289 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:25:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:25:45.166+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 1725 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:25:45.289 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:25:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:25:44.984+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:25:45.289 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:25:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:25:44.984+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:25:45.289 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:25:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:25:44.984+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:25:45.289 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:25:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:25:44.985+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:25:45.569 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:25:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:25:45.497+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:25:45.569 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:25:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:25:45.497+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:25:45.569 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:25:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:25:45.497+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:25:45.569 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:25:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:25:45.498+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:25:46.000 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:25:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:25:45.628+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:25:46.000 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:25:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:25:45.628+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:25:46.000 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:25:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:25:45.628+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:25:46.000 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:25:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:25:45.629+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:25:46.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:25:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:25:45.999+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:25:46.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:25:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:25:45.999+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:25:46.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:25:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:25:45.999+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:25:46.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:25:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:25:46.000+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:25:46.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:25:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:25:46.673+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:25:46.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:25:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:25:46.673+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:25:46.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:25:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:25:46.673+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:25:46.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:25:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:25:46.673+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:25:46.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:25:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:25:46.517+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:25:46.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:25:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:25:46.517+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:25:46.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:25:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:25:46.517+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:25:46.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:25:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:25:46.517+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:25:47.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:25:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:25:46.955+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:25:47.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:25:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:25:46.955+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:25:47.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:25:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:25:46.955+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:25:47.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:25:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:25:46.955+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:25:47.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:25:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:25:47.679+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:25:47.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:25:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:25:47.679+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:25:47.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:25:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:25:47.679+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:25:47.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:25:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:25:47.679+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:25:47.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:25:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:25:47.512+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:25:47.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:25:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:25:47.512+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:25:47.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:25:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:25:47.512+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:25:47.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:25:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:25:47.513+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:25:48.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:25:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:25:47.972+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:25:48.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:25:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:25:47.972+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:25:48.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:25:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:25:47.972+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:25:48.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:25:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:25:47.972+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:25:48.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:25:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:25:48.636+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:25:48.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:25:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:25:48.636+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:25:48.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:25:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:25:48.636+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:25:48.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:25:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:25:48.636+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:25:48.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:25:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:25:48.480+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:25:48.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:25:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:25:48.480+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:25:48.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:25:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:25:48.480+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:25:48.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:25:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:25:48.481+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:25:49.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:25:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:25:48.969+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:25:49.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:25:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:25:48.969+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:25:49.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:25:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:25:48.969+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:25:49.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:25:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:25:48.970+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:25:49.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:25:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:25:49.679+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:25:49.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:25:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:25:49.679+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:25:49.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:25:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:25:49.679+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:25:49.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:25:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:25:49.679+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:25:49.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:25:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:25:49.450+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:25:49.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:25:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:25:49.450+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:25:49.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:25:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:25:49.450+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:25:49.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:25:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:25:49.451+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:25:50.290 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:25:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:25:50.167+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 1816 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:25:50.290 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:25:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:25:49.959+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:25:50.290 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:25:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:25:49.959+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:25:50.290 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:25:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:25:49.959+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:25:50.290 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:25:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:25:49.960+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:25:50.569 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:25:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:25:50.490+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:25:50.569 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:25:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:25:50.490+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:25:50.569 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:25:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:25:50.490+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:25:50.569 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:25:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:25:50.490+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:25:50.975 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:25:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:25:50.687+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:25:50.975 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:25:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:25:50.687+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:25:50.975 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:25:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:25:50.687+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:25:50.975 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:25:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:25:50.688+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:25:51.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:25:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:25:50.974+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:25:51.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:25:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:25:50.974+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:25:51.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:25:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:25:50.974+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:25:51.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:25:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:25:50.975+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:25:51.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:25:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:25:51.525+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:25:51.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:25:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:25:51.525+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:25:51.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:25:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:25:51.525+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:25:51.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:25:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:25:51.525+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:25:51.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:25:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:25:51.689+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:25:51.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:25:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:25:51.689+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:25:51.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:25:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:25:51.689+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:25:51.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:25:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:25:51.690+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:25:52.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:25:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:25:51.966+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:25:52.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:25:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:25:51.966+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:25:52.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:25:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:25:51.966+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:25:52.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:25:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:25:51.967+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:25:52.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:25:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:25:52.493+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:25:52.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:25:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:25:52.493+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:25:52.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:25:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:25:52.493+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:25:52.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:25:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:25:52.493+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:25:52.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:25:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:25:52.714+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:25:52.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:25:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:25:52.714+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:25:52.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:25:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:25:52.714+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:25:52.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:25:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:25:52.715+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:25:53.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:25:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:25:52.926+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:25:53.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:25:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:25:52.926+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:25:53.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:25:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:25:52.926+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:25:53.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:25:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:25:52.927+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:25:53.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:25:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:25:53.466+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:25:53.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:25:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:25:53.466+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:25:53.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:25:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:25:53.466+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:25:53.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:25:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:25:53.467+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:25:54.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:25:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:25:53.961+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:25:54.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:25:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:25:53.961+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:25:54.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:25:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:25:53.961+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:25:54.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:25:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:25:53.962+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:25:54.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:25:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:25:53.728+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:25:54.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:25:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:25:53.728+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:25:54.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:25:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:25:53.728+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:25:54.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:25:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:25:53.729+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:25:54.775 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:25:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:25:54.447+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:25:54.775 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:25:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:25:54.447+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:25:54.775 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:25:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:25:54.447+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:25:54.775 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:25:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:25:54.447+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:25:55.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:25:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:25:54.951+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:25:55.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:25:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:25:54.951+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:25:55.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:25:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:25:54.951+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:25:55.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:25:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:25:54.952+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:25:55.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:25:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:25:54.773+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:25:55.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:25:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:25:54.773+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:25:55.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:25:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:25:54.773+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:25:55.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:25:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:25:54.774+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:25:55.421 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:25:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:25:55.166+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 1909 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:25:55.786 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:25:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:25:55.420+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:25:55.786 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:25:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:25:55.420+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:25:55.786 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:25:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:25:55.420+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:25:55.786 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:25:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:25:55.420+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:25:56.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:25:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:25:55.973+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:25:56.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:25:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:25:55.973+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:25:56.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:25:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:25:55.973+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:25:56.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:25:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:25:55.973+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:25:56.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:25:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:25:55.784+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:25:56.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:25:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:25:55.784+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:25:56.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:25:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:25:55.784+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:25:56.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:25:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:25:55.785+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:25:56.767 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:25:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:25:56.447+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:25:56.767 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:25:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:25:56.447+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:25:56.767 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:25:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:25:56.447+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:25:56.767 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:25:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:25:56.447+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:25:57.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:25:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:25:57.009+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:25:57.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:25:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:25:57.009+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:25:57.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:25:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:25:57.009+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:25:57.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:25:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:25:57.010+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:25:57.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:25:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:25:56.765+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:25:57.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:25:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:25:56.765+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:25:57.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:25:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:25:56.765+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:25:57.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:25:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:25:56.766+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:25:57.741 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:25:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:25:57.419+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:25:57.741 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:25:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:25:57.419+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:25:57.741 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:25:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:25:57.419+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:25:57.741 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:25:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:25:57.420+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:25:58.037 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:25:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:25:57.739+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:25:58.038 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:25:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:25:57.739+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:25:58.038 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:25:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:25:57.739+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:25:58.038 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:25:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:25:57.740+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:25:58.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:25:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:25:58.036+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:25:58.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:25:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:25:58.036+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:25:58.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:25:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:25:58.036+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:25:58.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:25:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:25:58.036+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:25:58.713 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:25:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:25:58.394+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:25:58.714 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:25:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:25:58.394+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:25:58.714 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:25:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:25:58.394+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:25:58.714 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:25:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:25:58.395+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:25:59.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:25:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:25:58.712+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:25:59.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:25:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:25:58.712+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:25:59.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:25:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:25:58.712+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:25:59.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:25:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:25:58.712+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:25:59.418 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:25:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:25:59.083+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:25:59.418 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:25:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:25:59.083+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:25:59.418 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:25:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:25:59.083+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:25:59.418 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:25:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:25:59.084+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:25:59.757 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:25:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:25:59.416+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:25:59.757 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:25:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:25:59.416+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:25:59.757 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:25:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:25:59.416+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:25:59.758 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:25:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:25:59.417+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:26:00.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:25:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:25:59.756+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:26:00.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:25:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:25:59.756+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:26:00.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:25:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:25:59.756+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:26:00.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:25:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:25:59.757+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:26:00.441 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:26:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:26:00.167+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 2001 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:26:00.441 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:26:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:26:00.069+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:26:00.441 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:26:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:26:00.069+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:26:00.441 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:26:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:26:00.069+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:26:00.441 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:26:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:26:00.069+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:26:00.749 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:26:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:26:00.439+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:26:00.750 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:26:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:26:00.439+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:26:00.750 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:26:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:26:00.439+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:26:00.750 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:26:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:26:00.439+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:26:01.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:26:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:26:00.748+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:26:01.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:26:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:26:00.748+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:26:01.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:26:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:26:00.748+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:26:01.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:26:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:26:00.749+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:26:01.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:26:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:26:01.091+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:26:01.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:26:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:26:01.091+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:26:01.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:26:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:26:01.091+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:26:01.448 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:26:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:26:01.091+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:26:01.771 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:26:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:26:01.446+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:26:01.771 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:26:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:26:01.446+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:26:01.771 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:26:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:26:01.446+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:26:01.772 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:26:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:26:01.446+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:26:02.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:26:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:26:01.771+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:26:02.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:26:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:26:01.771+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:26:02.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:26:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:26:01.771+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:26:02.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:26:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:26:01.771+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:26:02.466 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:26:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:26:02.085+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:26:02.467 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:26:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:26:02.086+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:26:02.467 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:26:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:26:02.086+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:26:02.467 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:26:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:26:02.086+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:26:02.739 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:26:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:26:02.466+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:26:02.739 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:26:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:26:02.466+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:26:02.739 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:26:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:26:02.466+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:26:02.739 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:26:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:26:02.466+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:26:03.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:26:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:26:02.738+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:26:03.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:26:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:26:02.738+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:26:03.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:26:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:26:02.738+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:26:03.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:26:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:26:02.738+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:26:03.480 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:26:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:26:03.108+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:26:03.480 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:26:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:26:03.108+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:26:03.480 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:26:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:26:03.108+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:26:03.480 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:26:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:26:03.108+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:26:03.782 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:26:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:26:03.480+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:26:03.782 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:26:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:26:03.480+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:26:03.782 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:26:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:26:03.480+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:26:03.782 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:26:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:26:03.480+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:26:04.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:26:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:26:03.781+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:26:04.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:26:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:26:03.781+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:26:04.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:26:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:26:03.781+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:26:04.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:26:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:26:03.782+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:26:04.506 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:26:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:26:04.120+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:26:04.506 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:26:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:26:04.120+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:26:04.506 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:26:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:26:04.120+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:26:04.507 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:26:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:26:04.121+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:26:04.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:26:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:26:04.505+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:26:04.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:26:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:26:04.505+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:26:04.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:26:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:26:04.505+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:26:04.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:26:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:26:04.506+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:26:05.120 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:26:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:26:04.827+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:26:05.121 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:26:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:26:04.827+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:26:05.121 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:26:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:26:04.827+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:26:05.121 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:26:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:26:04.828+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:26:05.508 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:26:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:26:05.169+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 2096 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:26:05.508 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:26:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:26:05.119+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:26:05.508 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:26:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:26:05.119+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:26:05.508 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:26:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:26:05.119+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:26:05.508 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:26:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:26:05.120+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:26:05.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:26:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:26:05.507+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:26:05.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:26:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:26:05.507+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:26:05.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:26:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:26:05.507+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:26:05.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:26:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:26:05.508+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:26:06.121 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:26:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:26:05.852+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:26:06.121 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:26:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:26:05.852+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:26:06.122 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:26:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:26:05.852+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:26:06.122 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:26:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:26:05.853+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:26:06.495 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:26:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:26:06.121+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:26:06.496 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:26:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:26:06.121+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:26:06.496 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:26:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:26:06.121+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:26:06.496 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:26:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:26:06.122+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:26:06.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:26:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:26:06.495+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:26:06.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:26:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:26:06.495+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:26:06.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:26:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:26:06.495+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:26:06.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:26:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:26:06.495+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:26:07.163 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:26:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:26:06.865+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:26:07.163 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:26:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:26:06.865+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:26:07.163 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:26:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:26:06.865+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:26:07.163 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:26:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:26:06.866+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:26:07.522 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:26:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:26:07.162+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:26:07.522 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:26:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:26:07.162+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:26:07.522 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:26:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:26:07.162+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:26:07.522 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:26:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:26:07.163+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:26:07.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:26:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:26:07.521+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:26:07.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:26:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:26:07.521+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:26:07.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:26:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:26:07.521+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:26:07.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:26:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:26:07.522+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:26:08.159 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:26:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:26:07.834+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:26:08.159 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:26:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:26:07.834+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:26:08.159 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:26:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:26:07.834+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:26:08.159 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:26:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:26:07.835+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:26:08.545 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:26:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:26:08.158+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:26:08.545 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:26:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:26:08.158+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:26:08.545 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:26:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:26:08.158+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:26:08.546 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:26:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:26:08.159+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:26:08.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:26:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:26:08.545+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:26:08.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:26:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:26:08.545+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:26:08.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:26:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:26:08.545+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:26:08.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:26:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:26:08.545+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:26:09.115 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:26:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:26:08.846+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:26:09.116 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:26:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:26:08.846+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:26:09.116 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:26:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:26:08.846+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:26:09.116 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:26:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:26:08.846+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:26:09.512 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:26:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:26:09.121+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:26:09.512 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:26:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:26:09.121+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:26:09.512 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:26:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:26:09.121+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:26:09.513 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:26:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:26:09.122+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:26:09.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:26:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:26:09.511+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:26:09.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:26:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:26:09.511+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:26:09.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:26:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:26:09.511+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:26:09.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:26:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:26:09.512+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:26:10.170 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:26:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:26:09.849+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:26:10.170 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:26:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:26:09.849+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:26:10.170 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:26:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:26:09.849+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:26:10.171 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:26:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:26:09.849+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:26:10.171 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:26:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:26:10.073+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:26:10.171 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:26:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:26:10.073+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:26:10.171 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:26:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:26:10.073+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:26:10.171 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:26:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:26:10.074+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:26:10.553 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:26:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:26:10.169+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 2189 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:26:10.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:26:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:26:10.553+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:26:10.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:26:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:26:10.553+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:26:10.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:26:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:26:10.553+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:26:10.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:26:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:26:10.553+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:26:11.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:26:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:26:11.025+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:26:11.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:26:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:26:11.025+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:26:11.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:26:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:26:11.025+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:26:11.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:26:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:26:11.026+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:26:11.319 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:26:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:26:10.864+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:26:11.319 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:26:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:26:10.864+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:26:11.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:26:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:26:10.864+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:26:11.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:26:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:26:10.865+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:26:11.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:26:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:26:11.510+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:26:11.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:26:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:26:11.510+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:26:11.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:26:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:26:11.510+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:26:11.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:26:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:26:11.510+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:26:12.319 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:26:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:26:11.822+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:26:12.319 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:26:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:26:11.822+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:26:12.319 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:26:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:26:11.822+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:26:12.319 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:26:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:26:11.823+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:26:12.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:26:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:26:11.996+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:26:12.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:26:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:26:11.996+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:26:12.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:26:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:26:11.996+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:26:12.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:26:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:26:11.996+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:26:12.790 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:26:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:26:12.527+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:26:12.790 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:26:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:26:12.527+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:26:12.790 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:26:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:26:12.527+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:26:12.791 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:26:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:26:12.528+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:26:13.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:26:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:26:12.789+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:26:13.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:26:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:26:12.789+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:26:13.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:26:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:26:12.789+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:26:13.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:26:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:26:12.790+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:26:13.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:26:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:26:12.969+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:26:13.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:26:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:26:12.969+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:26:13.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:26:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:26:12.969+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:26:13.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:26:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:26:12.970+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:26:13.781 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:26:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:26:13.492+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:26:13.782 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:26:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:26:13.492+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:26:13.782 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:26:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:26:13.492+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:26:13.782 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:26:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:26:13.493+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:26:14.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:26:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:26:13.781+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:26:14.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:26:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:26:13.781+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:26:14.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:26:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:26:13.781+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:26:14.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:26:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:26:13.781+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:26:14.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:26:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:26:14.019+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:26:14.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:26:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:26:14.019+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:26:14.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:26:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:26:14.019+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:26:14.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:26:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:26:14.019+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:26:14.765 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:26:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:26:14.471+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:26:14.765 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:26:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:26:14.471+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:26:14.765 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:26:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:26:14.471+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:26:14.765 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:26:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:26:14.471+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:26:15.027 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:26:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:26:14.764+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:26:15.027 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:26:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:26:14.764+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:26:15.028 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:26:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:26:14.764+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:26:15.028 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:26:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:26:14.765+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:26:15.293 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:26:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:26:15.170+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 2282 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:26:15.294 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:26:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:26:15.027+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:26:15.294 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:26:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:26:15.027+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:26:15.294 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:26:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:26:15.027+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:26:15.294 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:26:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:26:15.027+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:26:15.569 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:26:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:26:15.434+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:26:15.569 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:26:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:26:15.434+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:26:15.569 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:26:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:26:15.434+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:26:15.569 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:26:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:26:15.435+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:26:16.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:26:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:26:15.989+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:26:16.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:26:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:26:15.989+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:26:16.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:26:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:26:15.989+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:26:16.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:26:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:26:15.990+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:26:16.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:26:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:26:15.787+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:26:16.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:26:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:26:15.787+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:26:16.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:26:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:26:15.787+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:26:16.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:26:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:26:15.788+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:26:16.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:26:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:26:16.451+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:26:16.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:26:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:26:16.451+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:26:16.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:26:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:26:16.451+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:26:16.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:26:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:26:16.451+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:26:17.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:26:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:26:17.032+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:26:17.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:26:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:26:17.032+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:26:17.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:26:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:26:17.032+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:26:17.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:26:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:26:17.033+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:26:17.319 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:26:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:26:16.820+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:26:17.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:26:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:26:16.820+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:26:17.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:26:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:26:16.820+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:26:17.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:26:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:26:16.821+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:26:17.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:26:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:26:17.433+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:26:17.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:26:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:26:17.433+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:26:17.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:26:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:26:17.433+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:26:17.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:26:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:26:17.433+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:26:18.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:26:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:26:18.065+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:26:18.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:26:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:26:18.065+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:26:18.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:26:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:26:18.065+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:26:18.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:26:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:26:18.066+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:26:18.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:26:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:26:17.843+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:26:18.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:26:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:26:17.843+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:26:18.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:26:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:26:17.843+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:26:18.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:26:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:26:17.844+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:26:18.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:26:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:26:18.459+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:26:18.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:26:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:26:18.459+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:26:18.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:26:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:26:18.459+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:26:18.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:26:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:26:18.460+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:26:19.115 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:26:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:26:18.849+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:26:19.115 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:26:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:26:18.849+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:26:19.115 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:26:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:26:18.849+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:26:19.115 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:26:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:26:18.850+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:26:19.115 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:26:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:26:19.092+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:26:19.116 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:26:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:26:19.092+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:26:19.116 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:26:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:26:19.092+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:26:19.116 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:26:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:26:19.093+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:26:19.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:26:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:26:19.494+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:26:19.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:26:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:26:19.494+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:26:19.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:26:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:26:19.494+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:26:19.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:26:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:26:19.495+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:26:20.171 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:26:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:26:20.065+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:26:20.172 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:26:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:26:20.065+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:26:20.172 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:26:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:26:20.065+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:26:20.172 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:26:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:26:20.065+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:26:20.172 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:26:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:26:19.892+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:26:20.172 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:26:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:26:19.892+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:26:20.172 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:26:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:26:19.892+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:26:20.172 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:26:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:26:19.893+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:26:20.500 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:26:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:26:20.170+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 2373 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:26:20.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:26:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:26:20.499+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:26:20.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:26:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:26:20.499+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:26:20.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:26:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:26:20.499+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:26:20.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:26:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:26:20.500+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:26:21.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:26:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:26:21.044+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:26:21.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:26:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:26:21.044+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:26:21.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:26:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:26:21.044+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:26:21.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:26:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:26:21.045+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:26:21.319 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:26:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:26:20.885+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:26:21.319 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:26:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:26:20.885+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:26:21.319 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:26:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:26:20.885+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:26:21.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:26:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:26:20.885+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:26:21.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:26:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:26:21.487+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:26:21.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:26:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:26:21.487+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:26:21.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:26:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:26:21.487+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:26:21.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:26:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:26:21.487+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:26:22.234 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:26:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:26:21.919+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:26:22.235 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:26:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:26:21.919+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:26:22.235 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:26:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:26:21.919+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:26:22.235 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:26:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:26:21.920+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:26:22.235 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:26:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:26:22.041+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:26:22.235 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:26:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:26:22.041+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:26:22.235 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:26:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:26:22.041+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:26:22.235 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:26:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:26:22.042+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:26:22.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:26:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:26:22.489+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:26:22.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:26:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:26:22.489+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:26:22.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:26:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:26:22.489+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:26:22.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:26:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:26:22.490+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:26:23.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:26:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:26:23.041+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:26:23.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:26:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:26:23.041+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:26:23.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:26:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:26:23.041+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:26:23.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:26:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:26:23.042+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:26:23.319 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:26:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:26:22.888+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:26:23.319 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:26:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:26:22.888+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:26:23.319 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:26:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:26:22.888+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:26:23.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:26:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:26:22.889+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:26:23.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:26:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:26:23.539+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:26:23.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:26:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:26:23.539+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:26:23.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:26:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:26:23.539+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:26:23.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:26:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:26:23.539+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:26:24.319 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:26:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:26:23.904+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:26:24.319 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:26:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:26:23.904+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:26:24.319 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:26:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:26:23.904+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:26:24.319 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:26:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:26:23.905+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:26:24.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:26:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:26:24.031+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:26:24.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:26:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:26:24.031+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:26:24.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:26:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:26:24.031+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:26:24.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:26:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:26:24.032+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:26:24.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:26:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:26:24.541+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:26:24.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:26:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:26:24.541+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:26:24.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:26:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:26:24.541+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:26:24.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:26:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:26:24.542+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:26:25.183 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:26:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:26:25.171+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 2468 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:26:25.183 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:26:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:26:25.073+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:26:25.184 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:26:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:26:25.073+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:26:25.184 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:26:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:26:25.073+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:26:25.184 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:26:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:26:25.074+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:26:25.184 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:26:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:26:24.931+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:26:25.184 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:26:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:26:24.931+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:26:25.184 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:26:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:26:24.931+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:26:25.184 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:26:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:26:24.931+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:26:25.900 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:26:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:26:25.580+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:26:25.900 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:26:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:26:25.580+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:26:25.901 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:26:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:26:25.580+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:26:25.901 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:26:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:26:25.580+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:26:26.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:26:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:26:26.048+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:26:26.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:26:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:26:26.048+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:26:26.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:26:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:26:26.048+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:26:26.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:26:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:26:26.049+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:26:26.319 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:26:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:26:25.899+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:26:26.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:26:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:26:25.899+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:26:26.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:26:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:26:25.899+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:26:26.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:26:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:26:25.900+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:26:26.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:26:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:26:26.541+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:26:26.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:26:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:26:26.541+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:26:26.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:26:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:26:26.541+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:26:26.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:26:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:26:26.541+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:26:27.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:26:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:26:27.030+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:26:27.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:26:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:26:27.030+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:26:27.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:26:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:26:27.030+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:26:27.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:26:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:26:27.031+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:26:27.319 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:26:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:26:26.858+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:26:27.319 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:26:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:26:26.858+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:26:27.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:26:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:26:26.858+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:26:27.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:26:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:26:26.858+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:26:27.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:26:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:26:27.553+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:26:27.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:26:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:26:27.553+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:26:27.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:26:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:26:27.553+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:26:27.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:26:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:26:27.554+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:26:28.319 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:26:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:26:27.823+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:26:28.319 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:26:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:26:27.823+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:26:28.319 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:26:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:26:27.823+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:26:28.319 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:26:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:26:27.824+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:26:28.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:26:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:26:28.029+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:26:28.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:26:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:26:28.029+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:26:28.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:26:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:26:28.029+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:26:28.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:26:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:26:28.030+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:26:28.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:26:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:26:28.800+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:26:28.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:26:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:26:28.800+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:26:28.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:26:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:26:28.800+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:26:28.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:26:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:26:28.801+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:26:28.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:26:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:26:28.559+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:26:28.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:26:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:26:28.559+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:26:28.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:26:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:26:28.559+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:26:28.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:26:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:26:28.559+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:26:29.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:26:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:26:29.028+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:26:29.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:26:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:26:29.028+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:26:29.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:26:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:26:29.028+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:26:29.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:26:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:26:29.028+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:26:29.843 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:26:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:26:29.574+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:26:29.844 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:26:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:26:29.574+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:26:29.844 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:26:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:26:29.574+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:26:29.844 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:26:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:26:29.574+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:26:30.172 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:26:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:26:30.033+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:26:30.173 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:26:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:26:30.033+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:26:30.173 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:26:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:26:30.033+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:26:30.173 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:26:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:26:30.034+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:26:30.173 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:26:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:26:29.843+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:26:30.173 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:26:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:26:29.843+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:26:30.173 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:26:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:26:29.843+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:26:30.173 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:26:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:26:29.844+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:26:30.537 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:26:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:26:30.172+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 2562 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:26:30.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:26:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:26:30.536+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:26:30.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:26:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:26:30.536+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:26:30.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:26:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:26:30.536+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:26:30.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:26:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:26:30.537+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:26:31.319 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:26:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:26:30.827+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:26:31.319 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:26:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:26:30.827+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:26:31.319 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:26:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:26:30.827+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:26:31.319 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:26:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:26:30.828+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:26:31.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:26:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:26:30.994+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:26:31.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:26:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:26:30.994+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:26:31.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:26:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:26:30.994+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:26:31.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:26:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:26:30.994+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:26:31.818 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:26:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:26:31.532+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:26:31.818 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:26:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:26:31.532+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:26:31.818 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:26:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:26:31.532+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:26:31.818 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:26:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:26:31.532+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:26:32.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:26:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:26:31.997+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:26:32.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:26:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:26:31.997+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:26:32.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:26:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:26:31.997+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:26:32.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:26:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:26:31.997+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:26:32.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:26:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:26:31.817+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:26:32.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:26:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:26:31.817+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:26:32.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:26:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:26:31.817+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:26:32.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:26:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:26:31.818+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:26:32.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:26:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:26:32.538+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:26:32.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:26:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:26:32.538+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:26:32.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:26:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:26:32.538+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:26:32.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:26:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:26:32.538+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:26:33.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:26:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:26:33.022+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:26:33.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:26:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:26:33.022+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:26:33.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:26:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:26:33.022+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:26:33.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:26:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:26:33.022+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:26:33.319 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:26:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:26:32.844+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:26:33.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:26:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:26:32.844+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:26:33.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:26:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:26:32.844+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:26:33.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:26:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:26:32.845+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:26:33.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:26:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:26:33.513+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:26:33.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:26:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:26:33.513+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:26:33.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:26:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:26:33.513+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:26:33.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:26:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:26:33.513+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:26:34.319 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:26:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:26:33.837+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:26:34.319 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:26:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:26:33.837+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:26:34.319 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:26:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:26:33.837+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:26:34.319 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:26:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:26:33.837+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:26:34.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:26:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:26:34.037+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:26:34.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:26:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:26:34.037+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:26:34.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:26:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:26:34.037+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:26:34.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:26:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:26:34.038+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:26:34.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:26:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:26:34.473+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:26:34.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:26:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:26:34.473+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:26:34.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:26:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:26:34.473+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:26:34.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:26:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:26:34.474+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:26:35.173 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:26:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:26:34.833+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:26:35.173 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:26:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:26:34.833+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:26:35.173 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:26:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:26:34.833+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:26:35.174 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:26:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:26:34.834+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:26:35.174 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:26:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:26:35.038+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:26:35.174 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:26:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:26:35.038+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:26:35.174 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:26:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:26:35.038+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:26:35.174 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:26:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:26:35.038+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:26:35.432 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:26:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:26:35.172+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 2657 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:26:35.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:26:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:26:35.431+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:26:35.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:26:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:26:35.431+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:26:35.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:26:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:26:35.431+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:26:35.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:26:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:26:35.431+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:26:36.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:26:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:26:36.023+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:26:36.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:26:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:26:36.023+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:26:36.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:26:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:26:36.023+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:26:36.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:26:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:26:36.023+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:26:36.319 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:26:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:26:35.824+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:26:36.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:26:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:26:35.824+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:26:36.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:26:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:26:35.824+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:26:36.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:26:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:26:35.825+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:26:36.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:26:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:26:36.428+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:26:36.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:26:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:26:36.428+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:26:36.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:26:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:26:36.428+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:26:36.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:26:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:26:36.428+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:26:37.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:26:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:26:37.058+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:26:37.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:26:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:26:37.058+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:26:37.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:26:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:26:37.058+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:26:37.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:26:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:26:37.058+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:26:37.319 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:26:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:26:36.820+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:26:37.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:26:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:26:36.820+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:26:37.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:26:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:26:36.820+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:26:37.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:26:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:26:36.821+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:26:37.772 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:26:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:26:37.421+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:26:37.772 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:26:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:26:37.421+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:26:37.772 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:26:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:26:37.421+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:26:37.772 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:26:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:26:37.421+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:26:38.054 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:26:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:26:37.771+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:26:38.054 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:26:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:26:37.771+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:26:38.054 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:26:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:26:37.771+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:26:38.054 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:26:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:26:37.772+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:26:38.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:26:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:26:38.052+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:26:38.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:26:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:26:38.052+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:26:38.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:26:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:26:38.052+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:26:38.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:26:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:26:38.053+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:26:38.735 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:26:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:26:38.376+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:26:38.735 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:26:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:26:38.376+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:26:38.735 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:26:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:26:38.376+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:26:38.735 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:26:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:26:38.376+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:26:39.054 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:26:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:26:38.734+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:26:39.054 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:26:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:26:38.734+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:26:39.054 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:26:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:26:38.734+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:26:39.054 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:26:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:26:38.735+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:26:39.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:26:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:26:39.053+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:26:39.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:26:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:26:39.053+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:26:39.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:26:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:26:39.053+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:26:39.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:26:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:26:39.054+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:26:39.747 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:26:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:26:39.330+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:26:39.747 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:26:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:26:39.330+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:26:39.747 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:26:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:26:39.330+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:26:39.747 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:26:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:26:39.330+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:26:40.066 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:26:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:26:39.746+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:26:40.066 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:26:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:26:39.746+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:26:40.066 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:26:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:26:39.746+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:26:40.066 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:26:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:26:39.746+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:26:40.319 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:26:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:26:40.173+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 2748 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:26:40.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:26:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:26:40.065+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:26:40.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:26:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:26:40.065+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:26:40.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:26:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:26:40.065+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:26:40.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:26:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:26:40.066+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:26:40.766 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:26:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:26:40.354+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:26:40.766 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:26:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:26:40.354+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:26:40.766 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:26:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:26:40.354+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:26:40.767 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:26:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:26:40.354+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:26:41.065 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:26:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:26:40.765+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:26:41.065 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:26:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:26:40.765+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:26:41.065 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:26:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:26:40.765+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:26:41.065 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:26:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:26:40.766+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:26:41.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:26:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:26:41.064+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:26:41.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:26:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:26:41.064+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:26:41.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:26:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:26:41.064+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:26:41.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:26:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:26:41.065+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:26:41.647 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:26:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:26:41.394+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:26:41.648 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:26:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:26:41.394+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:26:41.648 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:26:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:26:41.394+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:26:41.648 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:26:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:26:41.395+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:26:42.025 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:26:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:26:41.716+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:26:42.026 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:26:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:26:41.716+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:26:42.026 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:26:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:26:41.716+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:26:42.026 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:26:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:26:41.717+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:26:42.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:26:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:26:42.025+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:26:42.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:26:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:26:42.025+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:26:42.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:26:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:26:42.025+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:26:42.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:26:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:26:42.026+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:26:42.681 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:26:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:26:42.380+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:26:42.681 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:26:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:26:42.380+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:26:42.681 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:26:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:26:42.380+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:26:42.681 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:26:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:26:42.381+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:26:42.981 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:26:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:26:42.680+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:26:42.981 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:26:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:26:42.680+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:26:42.981 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:26:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:26:42.680+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:26:42.982 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:26:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:26:42.681+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:26:43.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:26:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:26:42.981+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:26:43.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:26:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:26:42.981+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:26:43.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:26:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:26:42.981+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:26:43.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:26:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:26:42.981+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:26:43.682 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:26:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:26:43.426+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:26:43.683 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:26:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:26:43.426+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:26:43.683 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:26:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:26:43.426+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:26:43.683 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:26:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:26:43.427+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:26:43.991 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:26:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:26:43.681+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:26:43.992 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:26:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:26:43.681+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:26:43.992 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:26:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:26:43.681+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:26:43.992 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:26:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:26:43.682+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:26:44.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:26:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:26:43.991+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:26:44.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:26:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:26:43.991+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:26:44.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:26:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:26:43.991+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:26:44.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:26:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:26:43.991+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:26:44.642 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:26:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:26:44.438+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:26:44.642 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:26:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:26:44.438+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:26:44.642 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:26:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:26:44.438+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:26:44.642 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:26:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:26:44.439+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:26:44.982 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:26:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:26:44.641+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:26:44.982 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:26:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:26:44.641+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:26:44.982 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:26:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:26:44.641+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:26:44.982 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:26:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:26:44.642+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:26:45.298 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:26:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:26:45.174+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 2843 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:26:45.298 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:26:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:26:44.981+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:26:45.298 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:26:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:26:44.981+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:26:45.298 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:26:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:26:44.981+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:26:45.298 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:26:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:26:44.982+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:26:45.569 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:26:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:26:45.424+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:26:45.569 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:26:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:26:45.424+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:26:45.569 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:26:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:26:45.424+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:26:45.569 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:26:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:26:45.425+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:26:46.024 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:26:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:26:45.605+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:26:46.024 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:26:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:26:45.605+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:26:46.024 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:26:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:26:45.605+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:26:46.024 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:26:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:26:45.606+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:26:46.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:26:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:26:46.023+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:26:46.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:26:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:26:46.023+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:26:46.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:26:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:26:46.023+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:26:46.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:26:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:26:46.024+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:26:46.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:26:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:26:46.450+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:26:46.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:26:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:26:46.450+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:26:46.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:26:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:26:46.450+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:26:46.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:26:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:26:46.451+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:26:46.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:26:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:26:46.633+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:26:46.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:26:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:26:46.633+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:26:46.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:26:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:26:46.633+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:26:46.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:26:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:26:46.634+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:26:47.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:26:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:26:47.029+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:26:47.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:26:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:26:47.029+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:26:47.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:26:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:26:47.029+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:26:47.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:26:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:26:47.030+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:26:47.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:26:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:26:47.490+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:26:47.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:26:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:26:47.490+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:26:47.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:26:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:26:47.490+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:26:47.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:26:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:26:47.490+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:26:47.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:26:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:26:47.666+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:26:47.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:26:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:26:47.666+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:26:47.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:26:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:26:47.666+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:26:47.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:26:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:26:47.667+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:26:48.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:26:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:26:47.990+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:26:48.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:26:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:26:47.990+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:26:48.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:26:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:26:47.990+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:26:48.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:26:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:26:47.991+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:26:48.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:26:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:26:48.490+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:26:48.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:26:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:26:48.490+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:26:48.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:26:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:26:48.490+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:26:48.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:26:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:26:48.491+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:26:48.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:26:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:26:48.668+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:26:48.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:26:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:26:48.668+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:26:48.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:26:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:26:48.668+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:26:48.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:26:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:26:48.668+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:26:49.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:26:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:26:48.984+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:26:49.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:26:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:26:48.984+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:26:49.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:26:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:26:48.984+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:26:49.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:26:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:26:48.985+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:26:49.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:26:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:26:49.646+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:26:49.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:26:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:26:49.646+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:26:49.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:26:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:26:49.646+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:26:49.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:26:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:26:49.646+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:26:49.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:26:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:26:49.443+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:26:49.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:26:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:26:49.443+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:26:49.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:26:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:26:49.443+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:26:49.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:26:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:26:49.443+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:26:50.298 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:26:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:26:50.174+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 2934 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:26:50.299 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:26:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:26:50.015+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:26:50.299 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:26:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:26:50.015+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:26:50.299 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:26:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:26:50.015+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:26:50.299 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:26:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:26:50.016+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:26:50.569 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:26:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:26:50.474+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:26:50.569 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:26:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:26:50.474+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:26:50.569 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:26:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:26:50.474+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:26:50.569 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:26:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:26:50.474+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:26:50.989 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:26:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:26:50.611+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:26:50.990 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:26:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:26:50.611+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:26:50.990 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:26:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:26:50.611+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:26:50.990 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:26:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:26:50.612+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:26:51.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:26:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:26:50.989+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:26:51.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:26:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:26:50.989+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:26:51.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:26:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:26:50.989+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:26:51.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:26:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:26:50.989+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:26:51.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:26:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:26:51.660+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:26:51.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:26:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:26:51.660+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:26:51.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:26:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:26:51.660+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:26:51.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:26:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:26:51.660+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:26:51.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:26:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:26:51.438+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:26:51.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:26:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:26:51.438+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:26:51.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:26:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:26:51.438+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:26:51.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:26:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:26:51.438+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:26:52.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:26:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:26:52.034+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:26:52.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:26:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:26:52.034+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:26:52.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:26:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:26:52.034+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:26:52.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:26:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:26:52.035+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:26:52.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:26:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:26:52.466+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:26:52.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:26:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:26:52.466+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:26:52.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:26:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:26:52.466+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:26:52.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:26:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:26:52.466+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:26:52.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:26:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:26:52.661+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:26:52.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:26:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:26:52.661+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:26:52.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:26:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:26:52.661+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:26:52.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:26:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:26:52.661+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:26:53.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:26:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:26:53.013+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:26:53.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:26:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:26:53.013+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:26:53.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:26:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:26:53.013+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:26:53.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:26:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:26:53.014+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:26:53.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:26:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:26:53.471+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:26:53.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:26:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:26:53.471+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:26:53.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:26:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:26:53.471+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:26:53.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:26:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:26:53.471+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:26:53.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:26:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:26:53.673+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:26:53.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:26:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:26:53.673+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:26:53.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:26:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:26:53.673+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:26:53.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:26:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:26:53.674+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:26:54.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:26:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:26:54.007+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:26:54.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:26:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:26:54.007+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:26:54.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:26:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:26:54.007+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:26:54.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:26:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:26:54.008+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:26:54.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:26:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:26:54.626+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:26:54.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:26:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:26:54.626+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:26:54.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:26:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:26:54.626+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:26:54.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:26:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:26:54.627+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:26:54.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:26:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:26:54.454+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:26:54.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:26:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:26:54.454+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:26:54.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:26:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:26:54.454+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:26:54.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:26:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:26:54.454+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:26:55.297 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:26:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:26:55.174+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 3027 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:26:55.298 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:26:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:26:55.027+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:26:55.298 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:26:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:26:55.027+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:26:55.298 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:26:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:26:55.027+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:26:55.298 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:26:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:26:55.028+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:26:55.569 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:26:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:26:55.492+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:26:55.569 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:26:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:26:55.492+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:26:55.569 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:26:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:26:55.492+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:26:55.569 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:26:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:26:55.493+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:26:56.027 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:26:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:26:55.596+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:26:56.027 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:26:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:26:55.596+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:26:56.027 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:26:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:26:55.596+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:26:56.027 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:26:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:26:55.596+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:26:56.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:26:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:26:56.056+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:26:56.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:26:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:26:56.056+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:26:56.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:26:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:26:56.056+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:26:56.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:26:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:26:56.058+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:26:56.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:26:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:26:56.590+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:26:56.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:26:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:26:56.590+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:26:56.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:26:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:26:56.590+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:26:56.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:26:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:26:56.590+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:26:56.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:26:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:26:56.478+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:26:56.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:26:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:26:56.478+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:26:56.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:26:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:26:56.478+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:26:56.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:26:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:26:56.478+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:26:57.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:26:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:26:57.064+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:26:57.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:26:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:26:57.064+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:26:57.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:26:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:26:57.064+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:26:57.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:26:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:26:57.065+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:26:57.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:26:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:26:57.510+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:26:57.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:26:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:26:57.510+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:26:57.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:26:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:26:57.510+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:26:57.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:26:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:26:57.510+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:26:57.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:26:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:26:57.552+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:26:57.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:26:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:26:57.552+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:26:57.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:26:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:26:57.552+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:26:57.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:26:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:26:57.553+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:26:58.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:26:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:26:58.043+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:26:58.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:26:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:26:58.043+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:26:58.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:26:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:26:58.043+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:26:58.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:26:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:26:58.044+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:26:58.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:26:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:26:58.504+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:26:58.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:26:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:26:58.504+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:26:58.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:26:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:26:58.504+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:26:58.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:26:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:26:58.504+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:26:58.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:26:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:26:58.562+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:26:58.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:26:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:26:58.562+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:26:58.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:26:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:26:58.562+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:26:58.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:26:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:26:58.564+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:26:59.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:26:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:26:59.068+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:26:59.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:26:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:26:59.068+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:26:59.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:26:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:26:59.068+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:26:59.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:26:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:26:59.068+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:26:59.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:26:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:26:59.544+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:26:59.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:26:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:26:59.544+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:26:59.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:26:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:26:59.544+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:26:59.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:26:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:26:59.545+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:26:59.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:26:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:26:59.544+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:26:59.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:26:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:26:59.544+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:26:59.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:26:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:26:59.544+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:26:59.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:26:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:26:59.545+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:27:00.298 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:27:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:27:00.175+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 3118 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:27:00.299 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:27:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:27:00.038+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:27:00.299 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:27:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:27:00.038+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:27:00.299 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:27:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:27:00.038+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:27:00.299 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:27:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:27:00.039+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:27:00.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:27:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:27:00.582+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:27:00.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:27:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:27:00.582+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:27:00.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:27:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:27:00.582+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:27:00.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:27:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:27:00.583+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:27:00.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:27:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:27:00.556+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:27:00.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:27:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:27:00.556+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:27:00.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:27:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:27:00.556+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:27:00.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:27:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:27:00.556+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:27:01.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:27:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:27:01.011+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:27:01.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:27:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:27:01.011+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:27:01.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:27:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:27:01.011+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:27:01.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:27:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:27:01.012+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:27:01.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:27:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:27:01.559+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:27:01.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:27:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:27:01.559+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:27:01.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:27:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:27:01.559+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:27:01.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:27:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:27:01.559+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:27:01.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:27:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:27:01.563+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:27:01.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:27:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:27:01.563+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:27:01.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:27:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:27:01.563+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:27:01.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:27:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:27:01.563+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:27:02.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:27:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:27:02.051+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:27:02.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:27:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:27:02.051+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:27:02.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:27:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:27:02.051+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:27:02.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:27:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:27:02.052+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:27:02.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:27:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:27:02.564+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:27:02.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:27:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:27:02.564+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:27:02.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:27:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:27:02.564+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:27:02.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:27:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:27:02.565+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:27:02.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:27:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:27:02.520+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:27:02.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:27:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:27:02.520+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:27:02.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:27:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:27:02.520+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:27:02.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:27:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:27:02.520+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:27:03.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:27:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:27:03.016+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:27:03.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:27:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:27:03.016+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:27:03.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:27:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:27:03.016+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:27:03.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:27:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:27:03.017+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:27:03.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:27:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:27:03.533+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:27:03.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:27:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:27:03.533+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:27:03.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:27:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:27:03.533+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:27:03.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:27:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:27:03.534+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:27:03.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:27:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:27:03.570+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:27:03.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:27:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:27:03.570+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:27:03.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:27:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:27:03.570+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:27:03.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:27:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:27:03.570+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:27:04.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:27:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:27:04.013+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:27:04.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:27:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:27:04.013+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:27:04.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:27:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:27:04.013+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:27:04.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:27:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:27:04.014+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:27:04.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:27:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:27:04.558+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:27:04.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:27:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:27:04.558+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:27:04.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:27:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:27:04.558+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:27:04.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:27:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:27:04.558+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:27:04.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:27:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:27:04.564+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:27:04.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:27:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:27:04.564+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:27:04.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:27:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:27:04.564+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:27:04.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:27:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:27:04.565+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:27:05.302 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:27:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:27:05.177+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 3212 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:27:05.302 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:27:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:27:05.011+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:27:05.302 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:27:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:27:05.011+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:27:05.302 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:27:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:27:05.011+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:27:05.302 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:27:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:27:05.012+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:27:05.978 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:27:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:27:05.595+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:27:05.978 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:27:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:27:05.595+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:27:05.978 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:27:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:27:05.595+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:27:05.978 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:27:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:27:05.596+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:27:05.978 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:27:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:27:05.600+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:27:05.978 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:27:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:27:05.600+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:27:05.978 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:27:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:27:05.600+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:27:05.979 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:27:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:27:05.600+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:27:06.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:27:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:27:05.977+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:27:06.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:27:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:27:05.977+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:27:06.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:27:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:27:05.977+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:27:06.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:27:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:27:05.978+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:27:06.978 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:27:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:27:06.628+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:27:06.978 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:27:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:27:06.628+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:27:06.979 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:27:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:27:06.628+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:27:06.979 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:27:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:27:06.629+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:27:06.979 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:27:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:27:06.602+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:27:06.979 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:27:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:27:06.602+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:27:06.979 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:27:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:27:06.602+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:27:06.979 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:27:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:27:06.602+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:27:07.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:27:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:27:06.977+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:27:07.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:27:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:27:06.977+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:27:07.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:27:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:27:06.977+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:27:07.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:27:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:27:06.978+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:27:08.007 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:27:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:27:07.585+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:27:08.008 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:27:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:27:07.585+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:27:08.008 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:27:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:27:07.585+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:27:08.008 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:27:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:27:07.585+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:27:08.008 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:27:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:27:07.631+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:27:08.008 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:27:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:27:07.632+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:27:08.008 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:27:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:27:07.632+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:27:08.008 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:27:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:27:07.632+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:27:08.316 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:27:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:27:08.007+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:27:08.316 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:27:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:27:08.007+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:27:08.316 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:27:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:27:08.007+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:27:08.316 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:27:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:27:08.008+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:27:08.992 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:27:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:27:08.624+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:27:08.992 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:27:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:27:08.624+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:27:08.993 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:27:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:27:08.624+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:27:08.993 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:27:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:27:08.624+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:27:08.993 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:27:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:27:08.618+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:27:08.993 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:27:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:27:08.618+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:27:08.993 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:27:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:27:08.618+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:27:08.993 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:27:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:27:08.619+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:27:09.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:27:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:27:08.992+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:27:09.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:27:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:27:08.992+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:27:09.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:27:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:27:08.992+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:27:09.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:27:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:27:08.992+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:27:10.012 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:27:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:27:09.605+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:27:10.013 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:27:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:27:09.605+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:27:10.013 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:27:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:27:09.605+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:27:10.013 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:27:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:27:09.605+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:27:10.013 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:27:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:27:09.626+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:27:10.013 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:27:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:27:09.626+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:27:10.013 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:27:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:27:09.626+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:27:10.013 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:27:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:27:09.627+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:27:10.304 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:27:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:27:10.178+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 3305 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:27:10.304 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:27:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:27:10.012+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:27:10.304 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:27:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:27:10.012+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:27:10.304 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:27:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:27:10.012+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:27:10.304 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:27:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:27:10.013+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:27:11.055 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:27:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:27:10.615+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:27:11.056 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:27:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:27:10.615+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:27:11.056 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:27:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:27:10.615+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:27:11.056 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:27:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:27:10.616+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:27:11.056 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:27:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:27:10.633+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:27:11.056 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:27:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:27:10.633+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:27:11.056 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:27:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:27:10.633+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:27:11.056 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:27:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:27:10.634+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:27:11.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:27:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:27:11.055+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:27:11.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:27:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:27:11.055+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:27:11.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:27:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:27:11.055+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:27:11.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:27:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:27:11.056+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:27:12.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:27:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:27:11.643+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:27:12.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:27:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:27:11.643+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:27:12.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:27:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:27:11.643+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:27:12.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:27:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:27:11.643+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:27:12.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:27:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:27:11.665+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:27:12.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:27:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:27:11.665+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:27:12.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:27:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:27:11.665+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:27:12.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:27:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:27:11.666+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:27:12.569 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:27:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:27:12.076+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:27:12.569 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:27:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:27:12.076+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:27:12.569 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:27:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:27:12.076+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:27:12.569 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:27:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:27:12.077+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:27:13.062 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:27:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:27:12.708+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:27:13.063 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:27:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:27:12.708+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:27:13.063 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:27:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:27:12.708+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:27:13.063 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:27:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:27:12.709+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:27:13.063 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:27:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:27:12.682+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:27:13.063 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:27:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:27:12.682+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:27:13.063 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:27:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:27:12.682+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:27:13.063 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:27:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:27:12.683+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:27:13.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:27:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:27:13.062+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:27:13.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:27:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:27:13.062+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:27:13.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:27:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:27:13.062+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:27:13.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:27:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:27:13.063+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:27:13.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:27:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:27:13.702+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:27:13.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:27:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:27:13.702+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:27:13.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:27:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:27:13.702+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:27:13.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:27:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:27:13.702+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:27:13.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:27:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:27:13.726+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:27:13.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:27:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:27:13.726+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:27:13.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:27:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:27:13.726+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:27:13.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:27:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:27:13.727+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:27:14.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:27:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:27:14.058+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:27:14.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:27:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:27:14.058+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:27:14.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:27:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:27:14.058+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:27:14.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:27:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:27:14.059+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:27:15.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:27:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:27:14.715+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:27:15.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:27:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:27:14.715+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:27:15.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:27:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:27:14.715+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:27:15.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:27:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:27:14.716+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:27:15.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:27:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:27:14.654+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:27:15.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:27:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:27:14.654+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:27:15.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:27:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:27:14.654+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:27:15.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:27:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:27:14.655+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:27:15.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:27:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:27:15.179+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 3397 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:27:15.569 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:27:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:27:15.076+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:27:15.569 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:27:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:27:15.076+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:27:15.570 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:27:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:27:15.076+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:27:15.570 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:27:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:27:15.077+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:27:16.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:27:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:27:15.705+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:27:16.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:27:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:27:15.705+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:27:16.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:27:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:27:15.705+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:27:16.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:27:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:27:15.705+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:27:16.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:27:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:27:15.610+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:27:16.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:27:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:27:15.610+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:27:16.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:27:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:27:15.610+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:27:16.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:27:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:27:15.611+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:27:16.569 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:27:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:27:16.102+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:27:16.569 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:27:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:27:16.102+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:27:16.569 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:27:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:27:16.102+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:27:16.569 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:27:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:27:16.103+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:27:17.057 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:27:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:27:16.584+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:27:17.057 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:27:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:27:16.584+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:27:17.057 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:27:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:27:16.584+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:27:17.057 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:27:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:27:16.585+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:27:17.057 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:27:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:27:16.695+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:27:17.057 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:27:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:27:16.695+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:27:17.057 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:27:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:27:16.695+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:27:17.057 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:27:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:27:16.695+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:27:17.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:27:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:27:17.056+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:27:17.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:27:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:27:17.056+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:27:17.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:27:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:27:17.056+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:27:17.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:27:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:27:17.057+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:27:17.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:27:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:27:17.549+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:27:17.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:27:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:27:17.549+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:27:17.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:27:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:27:17.549+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:27:17.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:27:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:27:17.550+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:27:17.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:27:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:27:17.712+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:27:17.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:27:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:27:17.712+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:27:17.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:27:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:27:17.712+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:27:17.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:27:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:27:17.712+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:27:18.510 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:27:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:27:18.100+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:27:18.510 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:27:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:27:18.100+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:27:18.510 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:27:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:27:18.100+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:27:18.510 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:27:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:27:18.101+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:27:18.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:27:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:27:18.509+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:27:18.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:27:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:27:18.509+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:27:18.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:27:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:27:18.509+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:27:18.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:27:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:27:18.510+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:27:18.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:27:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:27:18.694+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:27:18.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:27:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:27:18.694+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:27:18.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:27:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:27:18.694+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:27:18.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:27:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:27:18.694+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:27:19.532 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:27:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:27:19.086+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:27:19.532 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:27:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:27:19.086+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:27:19.532 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:27:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:27:19.086+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:27:19.532 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:27:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:27:19.087+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:27:19.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:27:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:27:19.531+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:27:19.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:27:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:27:19.531+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:27:19.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:27:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:27:19.531+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:27:19.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:27:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:27:19.532+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:27:19.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:27:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:27:19.712+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:27:19.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:27:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:27:19.712+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:27:19.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:27:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:27:19.712+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:27:19.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:27:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:27:19.712+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:27:20.319 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:27:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:27:20.179+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 3489 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:27:20.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:27:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:27:20.059+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:27:20.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:27:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:27:20.059+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:27:20.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:27:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:27:20.059+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:27:20.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:27:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:27:20.060+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:27:20.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:27:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:27:20.532+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:27:20.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:27:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:27:20.532+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:27:20.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:27:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:27:20.532+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:27:20.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:27:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:27:20.533+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:27:20.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:27:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:27:20.702+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:27:20.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:27:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:27:20.702+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:27:20.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:27:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:27:20.702+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:27:20.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:27:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:27:20.702+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:27:21.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:27:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:27:21.038+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:27:21.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:27:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:27:21.038+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:27:21.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:27:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:27:21.038+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:27:21.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:27:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:27:21.039+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:27:21.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:27:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:27:21.556+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:27:21.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:27:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:27:21.556+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:27:21.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:27:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:27:21.556+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:27:21.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:27:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:27:21.557+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:27:21.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:27:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:27:21.730+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:27:21.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:27:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:27:21.730+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:27:21.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:27:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:27:21.730+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:27:21.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:27:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:27:21.731+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:27:22.562 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:27:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:27:22.075+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:27:22.562 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:27:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:27:22.075+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:27:22.562 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:27:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:27:22.075+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:27:22.562 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:27:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:27:22.076+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:27:22.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:27:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:27:22.561+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:27:22.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:27:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:27:22.561+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:27:22.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:27:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:27:22.561+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:27:22.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:27:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:27:22.562+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:27:22.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:27:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:27:22.710+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:27:22.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:27:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:27:22.710+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:27:22.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:27:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:27:22.710+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:27:22.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:27:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:27:22.711+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:27:23.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:27:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:27:23.030+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:27:23.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:27:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:27:23.030+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:27:23.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:27:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:27:23.030+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:27:23.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:27:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:27:23.031+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:27:23.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:27:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:27:23.712+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:27:23.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:27:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:27:23.712+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:27:23.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:27:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:27:23.712+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:27:23.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:27:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:27:23.713+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:27:23.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:27:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:27:23.537+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:27:23.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:27:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:27:23.537+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:27:23.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:27:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:27:23.537+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:27:23.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:27:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:27:23.537+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:27:24.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:27:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:27:24.003+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:27:24.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:27:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:27:24.003+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:27:24.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:27:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:27:24.003+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:27:24.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:27:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:27:24.004+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:27:24.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:27:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:27:24.740+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:27:24.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:27:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:27:24.740+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:27:24.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:27:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:27:24.740+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:27:24.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:27:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:27:24.741+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:27:24.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:27:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:27:24.543+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:27:24.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:27:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:27:24.543+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:27:24.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:27:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:27:24.543+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:27:24.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:27:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:27:24.544+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:27:25.302 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:27:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:27:25.180+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 3582 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:27:25.303 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:27:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:27:25.012+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:27:25.303 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:27:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:27:25.012+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:27:25.303 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:27:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:27:25.012+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:27:25.303 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:27:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:27:25.013+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:27:25.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:27:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:27:25.545+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:27:25.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:27:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:27:25.545+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:27:25.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:27:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:27:25.545+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:27:25.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:27:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:27:25.545+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:27:26.046 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:27:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:27:25.757+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:27:26.046 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:27:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:27:25.757+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:27:26.046 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:27:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:27:25.757+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:27:26.046 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:27:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:27:25.758+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:27:26.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:27:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:27:26.045+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:27:26.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:27:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:27:26.045+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:27:26.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:27:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:27:26.045+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:27:26.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:27:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:27:26.046+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:27:26.761 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:27:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:27:26.759+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:27:26.761 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:27:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:27:26.759+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:27:26.761 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:27:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:27:26.759+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:27:26.761 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:27:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:27:26.760+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:27:26.761 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:27:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:27:26.510+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:27:26.761 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:27:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:27:26.510+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:27:26.761 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:27:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:27:26.510+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:27:26.761 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:27:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:27:26.511+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:27:27.557 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:27:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:27:27.084+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:27:27.557 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:27:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:27:27.084+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:27:27.557 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:27:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:27:27.084+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:27:27.557 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:27:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:27:27.085+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:27:27.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:27:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:27:27.556+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:27:27.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:27:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:27:27.556+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:27:27.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:27:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:27:27.556+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:27:27.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:27:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:27:27.557+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:27:27.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:27:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:27:27.799+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:27:27.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:27:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:27:27.799+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:27:27.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:27:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:27:27.799+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:27:27.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:27:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:27:27.799+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:27:28.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:27:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:27:28.061+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:27:28.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:27:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:27:28.061+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:27:28.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:27:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:27:28.061+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:27:28.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:27:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:27:28.062+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:27:28.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:27:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:27:28.522+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:27:28.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:27:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:27:28.522+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:27:28.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:27:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:27:28.522+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:27:28.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:27:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:27:28.522+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:27:29.115 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:27:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:27:29.024+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:27:29.116 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:27:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:27:29.024+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:27:29.116 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:27:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:27:29.024+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:27:29.116 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:27:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:27:29.025+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:27:29.116 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:27:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:27:28.843+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:27:29.116 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:27:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:27:28.843+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:27:29.116 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:27:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:27:28.843+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:27:29.116 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:27:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:27:28.844+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:27:29.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:27:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:27:29.491+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:27:29.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:27:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:27:29.491+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:27:29.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:27:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:27:29.491+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:27:29.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:27:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:27:29.492+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:27:30.182 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:27:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:27:30.029+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:27:30.182 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:27:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:27:30.029+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:27:30.182 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:27:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:27:30.029+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:27:30.182 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:27:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:27:30.030+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:27:30.182 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:27:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:27:29.891+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:27:30.182 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:27:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:27:29.891+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:27:30.183 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:27:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:27:29.891+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:27:30.183 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:27:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:27:29.892+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:27:30.484 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:27:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:27:30.181+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 3676 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:27:30.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:27:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:27:30.484+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:27:30.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:27:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:27:30.484+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:27:30.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:27:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:27:30.484+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:27:30.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:27:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:27:30.485+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:27:31.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:27:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:27:31.055+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:27:31.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:27:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:27:31.055+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:27:31.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:27:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:27:31.055+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:27:31.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:27:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:27:31.056+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:27:31.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:27:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:27:30.884+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:27:31.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:27:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:27:30.884+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:27:31.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:27:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:27:30.884+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:27:31.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:27:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:27:30.885+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:27:31.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:27:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:27:31.486+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:27:31.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:27:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:27:31.486+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:27:31.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:27:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:27:31.486+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:27:31.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:27:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:27:31.487+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:27:32.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:27:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:27:32.097+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:27:32.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:27:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:27:32.097+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:27:32.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:27:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:27:32.097+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:27:32.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:27:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:27:32.098+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:27:32.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:27:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:27:31.934+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:27:32.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:27:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:27:31.934+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:27:32.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:27:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:27:31.934+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:27:32.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:27:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:27:31.934+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:27:32.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:27:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:27:32.513+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:27:32.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:27:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:27:32.513+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:27:32.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:27:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:27:32.513+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:27:32.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:27:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:27:32.513+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:27:33.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:27:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:27:32.890+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:27:33.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:27:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:27:32.890+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:27:33.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:27:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:27:32.890+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:27:33.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:27:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:27:32.891+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:27:33.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:27:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:27:33.117+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:27:33.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:27:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:27:33.117+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:27:33.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:27:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:27:33.117+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:27:33.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:27:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:27:33.118+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:27:33.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:27:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:27:33.467+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:27:33.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:27:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:27:33.467+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:27:33.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:27:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:27:33.467+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:27:33.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:27:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:27:33.468+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:27:34.159 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:27:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:27:33.845+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:27:34.160 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:27:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:27:33.845+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:27:34.160 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:27:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:27:33.845+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:27:34.160 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:27:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:27:33.845+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:27:34.487 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:27:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:27:34.159+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:27:34.487 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:27:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:27:34.159+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:27:34.487 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:27:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:27:34.159+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:27:34.487 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:27:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:27:34.160+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:27:34.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:27:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:27:34.486+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:27:34.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:27:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:27:34.486+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:27:34.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:27:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:27:34.486+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:27:34.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:27:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:27:34.487+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:27:35.153 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:27:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:27:34.860+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:27:35.153 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:27:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:27:34.860+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:27:35.153 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:27:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:27:34.860+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:27:35.154 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:27:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:27:34.860+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:27:35.465 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:27:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:27:35.182+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 3769 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:27:35.466 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:27:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:27:35.152+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:27:35.466 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:27:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:27:35.152+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:27:35.466 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:27:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:27:35.152+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:27:35.466 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:27:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:27:35.153+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:27:35.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:27:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:27:35.464+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:27:35.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:27:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:27:35.464+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:27:35.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:27:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:27:35.464+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:27:35.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:27:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:27:35.464+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:27:36.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:27:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:27:36.141+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:27:36.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:27:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:27:36.141+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:27:36.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:27:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:27:36.142+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:27:36.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:27:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:27:36.142+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:27:36.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:27:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:27:35.896+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:27:36.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:27:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:27:35.896+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:27:36.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:27:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:27:35.896+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:27:36.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:27:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:27:35.897+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:27:36.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:27:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:27:36.436+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:27:36.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:27:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:27:36.436+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:27:36.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:27:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:27:36.436+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:27:36.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:27:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:27:36.437+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:27:37.142 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:27:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:27:36.858+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:27:37.142 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:27:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:27:36.858+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:27:37.142 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:27:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:27:36.858+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:27:37.143 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:27:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:27:36.859+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:27:37.486 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:27:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:27:37.141+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:27:37.486 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:27:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:27:37.141+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:27:37.486 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:27:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:27:37.141+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:27:37.487 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:27:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:27:37.142+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:27:37.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:27:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:27:37.486+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:27:37.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:27:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:27:37.486+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:27:37.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:27:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:27:37.486+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:27:37.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:27:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:27:37.487+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:27:38.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:27:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:27:38.124+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:27:38.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:27:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:27:38.124+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:27:38.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:27:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:27:38.124+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:27:38.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:27:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:27:38.125+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:27:38.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:27:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:27:37.901+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:27:38.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:27:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:27:37.901+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:27:38.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:27:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:27:37.901+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:27:38.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:27:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:27:37.901+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:27:38.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:27:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:27:38.446+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:27:38.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:27:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:27:38.446+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:27:38.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:27:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:27:38.446+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:27:38.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:27:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:27:38.447+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:27:39.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:27:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:27:39.172+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:27:39.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:27:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:27:39.172+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:27:39.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:27:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:27:39.172+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:27:39.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:27:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:27:39.172+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:27:39.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:27:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:27:38.940+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:27:39.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:27:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:27:38.940+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:27:39.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:27:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:27:38.940+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:27:39.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:27:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:27:38.941+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:27:39.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:27:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:27:39.494+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:27:39.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:27:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:27:39.494+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:27:39.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:27:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:27:39.494+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:27:39.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:27:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:27:39.495+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:27:40.305 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:27:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:27:40.182+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 3862 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:27:40.305 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:27:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:27:39.977+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:27:40.305 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:27:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:27:39.977+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:27:40.306 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:27:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:27:39.977+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:27:40.306 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:27:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:27:39.977+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:27:40.306 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:27:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:27:40.190+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:27:40.306 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:27:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:27:40.190+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:27:40.306 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:27:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:27:40.190+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:27:40.306 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:27:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:27:40.192+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:27:40.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:27:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:27:40.523+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:27:40.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:27:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:27:40.523+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:27:40.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:27:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:27:40.523+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:27:40.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:27:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:27:40.524+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:27:41.207 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:27:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:27:40.928+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:27:41.207 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:27:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:27:40.928+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:27:41.207 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:27:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:27:40.928+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:27:41.207 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:27:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:27:40.929+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:27:41.553 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:27:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:27:41.206+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:27:41.553 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:27:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:27:41.206+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:27:41.553 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:27:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:27:41.206+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:27:41.553 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:27:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:27:41.207+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:27:41.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:27:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:27:41.552+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:27:41.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:27:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:27:41.552+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:27:41.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:27:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:27:41.552+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:27:41.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:27:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:27:41.553+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:27:42.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:27:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:27:42.174+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:27:42.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:27:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:27:42.174+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:27:42.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:27:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:27:42.174+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:27:42.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:27:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:27:42.175+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:27:42.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:27:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:27:41.972+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:27:42.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:27:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:27:41.972+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:27:42.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:27:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:27:41.972+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:27:42.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:27:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:27:41.972+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:27:42.986 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:27:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:27:42.587+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:27:42.986 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:27:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:27:42.587+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:27:42.986 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:27:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:27:42.587+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:27:42.986 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:27:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:27:42.588+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:27:43.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:27:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:27:43.131+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:27:43.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:27:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:27:43.131+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:27:43.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:27:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:27:43.131+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:27:43.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:27:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:27:43.132+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:27:43.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:27:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:27:42.985+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:27:43.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:27:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:27:42.985+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:27:43.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:27:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:27:42.985+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:27:43.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:27:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:27:42.986+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:27:43.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:27:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:27:43.603+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:27:43.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:27:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:27:43.603+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:27:43.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:27:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:27:43.603+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:27:43.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:27:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:27:43.604+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:27:44.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:27:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:27:44.088+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:27:44.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:27:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:27:44.088+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:27:44.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:27:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:27:44.088+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:27:44.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:27:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:27:44.089+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:27:44.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:27:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:27:44.010+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:27:44.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:27:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:27:44.010+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:27:44.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:27:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:27:44.010+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:27:44.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:27:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:27:44.011+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:27:44.966 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:27:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:27:44.640+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:27:44.966 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:27:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:27:44.640+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:27:44.966 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:27:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:27:44.640+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:27:44.966 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:27:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:27:44.640+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:27:45.306 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:27:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:27:45.183+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 3955 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:27:45.306 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:27:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:27:44.965+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:27:45.306 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:27:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:27:44.965+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:27:45.306 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:27:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:27:44.965+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:27:45.306 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:27:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:27:44.966+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:27:45.307 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:27:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:27:45.137+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:27:45.307 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:27:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:27:45.137+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:27:45.307 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:27:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:27:45.137+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:27:45.307 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:27:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:27:45.137+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:27:45.934 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:27:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:27:45.610+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:27:45.934 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:27:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:27:45.610+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:27:45.934 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:27:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:27:45.610+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:27:45.934 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:27:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:27:45.611+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:27:46.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:27:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:27:46.121+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:27:46.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:27:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:27:46.121+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:27:46.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:27:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:27:46.121+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:27:46.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:27:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:27:46.122+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:27:46.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:27:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:27:45.933+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:27:46.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:27:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:27:45.933+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:27:46.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:27:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:27:45.933+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:27:46.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:27:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:27:45.933+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:27:46.955 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:27:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:27:46.659+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:27:46.955 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:27:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:27:46.659+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:27:46.955 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:27:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:27:46.659+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:27:46.955 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:27:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:27:46.660+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:27:47.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:27:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:27:47.150+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:27:47.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:27:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:27:47.150+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:27:47.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:27:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:27:47.150+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:27:47.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:27:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:27:47.151+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:27:47.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:27:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:27:46.954+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:27:47.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:27:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:27:46.954+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:27:47.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:27:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:27:46.954+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:27:47.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:27:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:27:46.955+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:27:48.001 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:27:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:27:47.614+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:27:48.001 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:27:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:27:47.614+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:27:48.002 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:27:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:27:47.614+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:27:48.002 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:27:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:27:47.615+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:27:48.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:27:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:27:48.106+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:27:48.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:27:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:27:48.106+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:27:48.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:27:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:27:48.106+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:27:48.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:27:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:27:48.106+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:27:48.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:27:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:27:48.001+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:27:48.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:27:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:27:48.001+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:27:48.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:27:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:27:48.001+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:27:48.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:27:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:27:48.001+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:27:49.042 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:27:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:27:48.603+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:27:49.043 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:27:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:27:48.603+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:27:49.043 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:27:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:27:48.603+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:27:49.043 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:27:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:27:48.604+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:27:49.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:27:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:27:49.072+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:27:49.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:27:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:27:49.072+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:27:49.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:27:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:27:49.072+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:27:49.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:27:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:27:49.073+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:27:49.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:27:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:27:49.042+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:27:49.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:27:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:27:49.042+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:27:49.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:27:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:27:49.042+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:27:49.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:27:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:27:49.042+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:27:49.999 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:27:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:27:49.591+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:27:49.999 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:27:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:27:49.591+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:27:49.999 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:27:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:27:49.591+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:27:49.999 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:27:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:27:49.592+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:27:50.307 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:27:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:27:50.184+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 4048 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:27:50.307 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:27:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:27:50.101+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:27:50.307 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:27:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:27:50.101+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:27:50.307 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:27:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:27:50.101+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:27:50.307 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:27:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:27:50.102+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:27:50.307 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:27:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:27:49.998+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:27:50.308 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:27:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:27:49.998+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:27:50.308 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:27:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:27:49.998+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:27:50.308 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:27:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:27:49.999+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:27:51.011 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:27:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:27:50.586+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:27:51.011 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:27:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:27:50.586+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:27:51.012 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:27:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:27:50.586+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:27:51.012 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:27:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:27:50.587+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:27:51.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:27:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:27:51.010+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:27:51.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:27:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:27:51.010+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:27:51.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:27:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:27:51.010+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:27:51.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:27:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:27:51.011+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:27:51.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:27:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:27:51.057+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:27:51.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:27:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:27:51.057+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:27:51.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:27:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:27:51.057+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:27:51.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:27:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:27:51.058+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:27:51.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:27:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:27:51.543+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:27:51.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:27:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:27:51.543+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:27:51.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:27:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:27:51.543+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:27:51.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:27:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:27:51.544+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:27:52.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:27:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:27:52.016+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:27:52.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:27:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:27:52.016+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:27:52.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:27:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:27:52.016+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:27:52.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:27:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:27:52.017+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:27:52.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:27:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:27:52.080+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:27:52.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:27:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:27:52.080+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:27:52.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:27:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:27:52.080+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:27:52.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:27:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:27:52.081+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:27:52.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:27:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:27:52.526+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:27:52.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:27:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:27:52.526+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:27:52.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:27:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:27:52.526+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:27:52.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:27:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:27:52.526+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:27:53.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:27:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:27:53.049+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:27:53.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:27:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:27:53.049+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:27:53.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:27:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:27:53.049+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:27:53.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:27:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:27:53.050+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:27:53.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:27:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:27:52.967+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:27:53.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:27:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:27:52.967+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:27:53.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:27:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:27:52.967+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:27:53.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:27:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:27:52.967+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:27:53.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:27:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:27:53.480+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:27:53.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:27:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:27:53.480+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:27:53.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:27:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:27:53.480+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:27:53.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:27:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:27:53.480+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:27:54.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:27:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:27:54.056+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:27:54.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:27:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:27:54.056+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:27:54.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:27:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:27:54.056+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:27:54.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:27:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:27:54.056+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:27:54.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:27:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:27:53.946+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:27:54.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:27:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:27:53.946+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:27:54.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:27:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:27:53.946+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:27:54.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:27:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:27:53.947+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:27:54.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:27:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:27:54.523+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:27:54.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:27:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:27:54.523+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:27:54.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:27:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:27:54.523+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:27:54.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:27:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:27:54.523+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:27:55.308 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:27:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:27:55.184+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 4140 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:27:55.309 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:27:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:27:54.984+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:27:55.309 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:27:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:27:54.984+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:27:55.309 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:27:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:27:54.984+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:27:55.309 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:27:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:27:54.985+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:27:55.309 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:27:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:27:55.051+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:27:55.309 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:27:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:27:55.051+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:27:55.309 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:27:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:27:55.051+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:27:55.309 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:27:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:27:55.052+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:27:55.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:27:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:27:55.511+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:27:55.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:27:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:27:55.511+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:27:55.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:27:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:27:55.511+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:27:55.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:27:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:27:55.511+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:27:56.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:27:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:27:56.049+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:27:56.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:27:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:27:56.049+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:27:56.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:27:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:27:56.049+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:27:56.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:27:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:27:56.049+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:27:56.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:27:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:27:55.953+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:27:56.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:27:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:27:55.953+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:27:56.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:27:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:27:55.953+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:27:56.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:27:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:27:55.953+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:27:56.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:27:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:27:56.531+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:27:56.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:27:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:27:56.531+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:27:56.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:27:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:27:56.531+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:27:56.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:27:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:27:56.532+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:27:57.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:27:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:27:57.095+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:27:57.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:27:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:27:57.095+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:27:57.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:27:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:27:57.095+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:27:57.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:27:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:27:57.096+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:27:57.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:27:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:27:56.942+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:27:57.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:27:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:27:56.942+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:27:57.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:27:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:27:56.942+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:27:57.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:27:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:27:56.943+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:27:57.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:27:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:27:57.503+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:27:57.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:27:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:27:57.503+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:27:57.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:27:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:27:57.503+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:27:57.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:27:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:27:57.504+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:27:58.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:27:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:27:58.053+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:27:58.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:27:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:27:58.054+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:27:58.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:27:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:27:58.054+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:27:58.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:27:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:27:58.054+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:27:58.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:27:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:27:57.954+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:27:58.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:27:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:27:57.954+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:27:58.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:27:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:27:57.954+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:27:58.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:27:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:27:57.955+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:27:58.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:27:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:27:58.468+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:27:58.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:27:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:27:58.468+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:27:58.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:27:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:27:58.468+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:27:58.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:27:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:27:58.468+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:27:59.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:27:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:27:59.014+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:27:59.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:27:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:27:59.014+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:27:59.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:27:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:27:59.014+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:27:59.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:27:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:27:59.015+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:27:59.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:27:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:27:58.984+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:27:59.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:27:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:27:58.984+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:27:59.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:27:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:27:58.984+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:27:59.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:27:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:27:58.984+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:27:59.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:27:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:27:59.494+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:27:59.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:27:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:27:59.494+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:27:59.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:27:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:27:59.494+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:27:59.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:27:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:27:59.495+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:28:00.198 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:28:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:28:00.184+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 4232 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:28:00.198 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:27:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:27:59.943+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:28:00.198 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:27:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:27:59.943+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:28:00.198 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:27:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:27:59.943+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:28:00.198 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:27:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:27:59.943+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:28:00.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:28:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:28:00.034+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:28:00.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:28:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:28:00.034+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:28:00.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:28:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:28:00.034+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:28:00.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:28:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:28:00.035+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:28:00.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:28:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:28:00.538+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:28:00.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:28:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:28:00.538+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:28:00.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:28:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:28:00.538+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:28:00.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:28:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:28:00.539+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:28:01.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:28:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:28:01.005+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:28:01.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:28:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:28:01.005+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:28:01.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:28:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:28:01.005+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:28:01.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:28:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:28:01.006+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:28:01.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:28:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:28:00.992+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:28:01.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:28:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:28:00.992+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:28:01.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:28:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:28:00.992+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:28:01.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:28:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:28:00.992+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:28:01.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:28:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:28:01.534+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:28:01.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:28:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:28:01.534+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:28:01.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:28:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:28:01.534+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:28:01.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:28:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:28:01.534+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:28:02.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:28:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:28:02.055+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:28:02.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:28:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:28:02.055+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:28:02.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:28:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:28:02.055+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:28:02.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:28:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:28:02.056+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:28:02.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:28:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:28:01.985+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:28:02.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:28:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:28:01.985+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:28:02.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:28:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:28:01.985+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:28:02.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:28:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:28:01.985+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:28:02.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:28:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:28:02.495+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:28:02.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:28:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:28:02.495+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:28:02.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:28:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:28:02.495+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:28:02.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:28:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:28:02.495+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:28:03.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:28:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:28:03.061+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:28:03.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:28:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:28:03.061+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:28:03.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:28:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:28:03.061+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:28:03.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:28:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:28:03.062+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:28:03.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:28:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:28:03.002+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:28:03.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:28:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:28:03.002+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:28:03.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:28:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:28:03.002+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:28:03.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:28:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:28:03.003+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:28:03.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:28:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:28:03.449+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:28:03.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:28:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:28:03.449+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:28:03.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:28:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:28:03.449+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:28:03.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:28:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:28:03.450+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:28:04.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:28:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:28:04.069+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:28:04.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:28:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:28:04.069+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:28:04.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:28:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:28:04.069+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:28:04.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:28:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:28:04.070+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:28:04.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:28:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:28:03.976+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:28:04.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:28:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:28:03.976+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:28:04.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:28:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:28:03.976+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:28:04.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:28:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:28:03.977+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:28:04.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:28:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:28:04.492+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:28:04.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:28:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:28:04.492+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:28:04.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:28:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:28:04.492+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:28:04.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:28:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:28:04.493+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:28:05.310 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:28:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:28:05.186+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 4330 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:28:05.310 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:28:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:28:05.006+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:28:05.310 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:28:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:28:05.006+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:28:05.310 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:28:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:28:05.006+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:28:05.311 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:28:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:28:05.006+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:28:05.311 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:28:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:28:05.076+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:28:05.311 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:28:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:28:05.076+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:28:05.311 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:28:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:28:05.076+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:28:05.311 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:28:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:28:05.077+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:28:05.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:28:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:28:05.459+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:28:05.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:28:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:28:05.459+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:28:05.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:28:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:28:05.459+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:28:05.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:28:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:28:05.460+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:28:06.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:28:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:28:06.110+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:28:06.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:28:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:28:06.110+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:28:06.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:28:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:28:06.110+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:28:06.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:28:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:28:06.110+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:28:06.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:28:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:28:06.017+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:28:06.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:28:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:28:06.018+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:28:06.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:28:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:28:06.018+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:28:06.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:28:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:28:06.018+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:28:06.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:28:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:28:06.500+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:28:06.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:28:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:28:06.500+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:28:06.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:28:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:28:06.500+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:28:06.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:28:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:28:06.500+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:28:07.116 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:28:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:28:07.032+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:28:07.116 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:28:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:28:07.032+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:28:07.116 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:28:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:28:07.032+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:28:07.117 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:28:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:28:07.033+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:28:07.464 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:28:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:28:07.115+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:28:07.465 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:28:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:28:07.116+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:28:07.465 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:28:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:28:07.116+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:28:07.465 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:28:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:28:07.117+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:28:07.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:28:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:28:07.464+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:28:07.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:28:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:28:07.464+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:28:07.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:28:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:28:07.464+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:28:07.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:28:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:28:07.464+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:28:08.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:28:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:28:07.994+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:28:08.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:28:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:28:07.994+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:28:08.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:28:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:28:07.994+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:28:08.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:28:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:28:07.995+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:28:08.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:28:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:28:08.131+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:28:08.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:28:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:28:08.132+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:28:08.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:28:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:28:08.132+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:28:08.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:28:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:28:08.133+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:28:08.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:28:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:28:08.438+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:28:08.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:28:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:28:08.438+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:28:08.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:28:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:28:08.438+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:28:08.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:28:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:28:08.439+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:28:09.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:28:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:28:09.168+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:28:09.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:28:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:28:09.168+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:28:09.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:28:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:28:09.168+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:28:09.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:28:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:28:09.169+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:28:09.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:28:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:28:09.006+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:28:09.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:28:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:28:09.006+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:28:09.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:28:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:28:09.006+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:28:09.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:28:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:28:09.006+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:28:09.755 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:28:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:28:09.443+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:28:09.755 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:28:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:28:09.443+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:28:09.755 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:28:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:28:09.443+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:28:09.755 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:28:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:28:09.444+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:28:10.310 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:28:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:28:10.187+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 4425 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:28:10.310 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:28:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:28:10.168+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:28:10.311 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:28:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:28:10.168+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:28:10.311 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:28:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:28:10.168+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:28:10.311 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:28:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:28:10.169+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:28:10.311 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:28:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:28:09.993+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:28:10.311 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:28:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:28:09.993+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:28:10.311 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:28:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:28:09.993+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:28:10.311 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:28:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:28:09.993+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:28:10.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:28:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:28:10.440+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:28:10.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:28:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:28:10.440+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:28:10.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:28:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:28:10.440+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:28:10.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:28:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:28:10.440+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:28:11.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:28:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:28:11.042+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:28:11.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:28:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:28:11.042+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:28:11.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:28:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:28:11.042+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:28:11.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:28:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:28:11.042+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:28:11.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:28:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:28:11.193+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:28:11.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:28:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:28:11.193+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:28:11.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:28:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:28:11.193+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:28:11.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:28:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:28:11.194+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:28:11.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:28:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:28:11.458+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:28:11.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:28:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:28:11.458+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:28:11.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:28:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:28:11.458+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:28:11.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:28:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:28:11.458+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:28:12.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:28:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:28:12.200+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:28:12.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:28:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:28:12.200+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:28:12.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:28:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:28:12.200+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:28:12.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:28:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:28:12.200+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:28:12.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:28:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:28:12.044+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:28:12.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:28:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:28:12.044+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:28:12.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:28:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:28:12.044+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:28:12.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:28:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:28:12.044+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:28:12.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:28:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:28:12.485+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:28:12.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:28:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:28:12.485+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:28:12.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:28:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:28:12.485+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:28:12.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:28:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:28:12.486+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:28:13.443 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:28:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:28:13.230+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:28:13.443 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:28:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:28:13.230+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:28:13.443 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:28:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:28:13.230+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:28:13.443 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:28:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:28:13.231+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:28:13.443 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:28:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:28:13.081+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:28:13.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:28:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:28:13.081+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:28:13.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:28:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:28:13.081+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:28:13.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:28:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:28:13.081+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:28:13.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:28:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:28:13.442+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:28:13.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:28:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:28:13.442+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:28:13.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:28:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:28:13.442+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:28:13.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:28:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:28:13.442+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:28:14.453 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:28:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:28:14.261+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:28:14.453 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:28:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:28:14.261+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:28:14.453 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:28:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:28:14.261+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:28:14.454 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:28:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:28:14.262+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:28:14.454 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:28:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:28:14.073+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:28:14.454 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:28:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:28:14.073+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:28:14.454 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:28:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:28:14.073+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:28:14.454 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:28:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:28:14.073+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:28:14.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:28:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:28:14.452+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:28:14.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:28:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:28:14.452+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:28:14.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:28:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:28:14.452+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:28:14.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:28:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:28:14.453+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:28:15.470 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:28:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:28:15.188+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 4523 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:28:15.470 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:28:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:28:15.281+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:28:15.470 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:28:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:28:15.282+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:28:15.470 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:28:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:28:15.282+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:28:15.470 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:28:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:28:15.283+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:28:15.470 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:28:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:28:15.111+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:28:15.471 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:28:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:28:15.111+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:28:15.471 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:28:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:28:15.111+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:28:15.471 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:28:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:28:15.111+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:28:15.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:28:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:28:15.469+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:28:15.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:28:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:28:15.469+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:28:15.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:28:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:28:15.469+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:28:15.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:28:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:28:15.469+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:28:16.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:28:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:28:16.098+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:28:16.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:28:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:28:16.098+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:28:16.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:28:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:28:16.098+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:28:16.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:28:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:28:16.099+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:28:16.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:28:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:28:16.318+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:28:16.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:28:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:28:16.318+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:28:16.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:28:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:28:16.318+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:28:16.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:28:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:28:16.319+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:28:16.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:28:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:28:16.428+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:28:16.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:28:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:28:16.428+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:28:16.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:28:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:28:16.428+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:28:16.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:28:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:28:16.429+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:28:17.470 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:28:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:28:17.326+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:28:17.470 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:28:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:28:17.326+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:28:17.470 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:28:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:28:17.326+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:28:17.471 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:28:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:28:17.327+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:28:17.471 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:28:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:28:17.123+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:28:17.471 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:28:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:28:17.123+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:28:17.471 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:28:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:28:17.123+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:28:17.471 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:28:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:28:17.124+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:28:17.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:28:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:28:17.469+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:28:17.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:28:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:28:17.469+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:28:17.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:28:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:28:17.469+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:28:17.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:28:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:28:17.470+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:28:18.353 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:28:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:28:18.080+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:28:18.353 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:28:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:28:18.080+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:28:18.353 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:28:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:28:18.080+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:28:18.353 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:28:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:28:18.080+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:28:18.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:28:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:28:18.352+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:28:18.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:28:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:28:18.352+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:28:18.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:28:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:28:18.352+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:28:18.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:28:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:28:18.353+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:28:18.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:28:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:28:18.479+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:28:18.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:28:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:28:18.479+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:28:18.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:28:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:28:18.479+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:28:18.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:28:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:28:18.480+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:28:19.442 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:28:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:28:19.318+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:28:19.442 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:28:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:28:19.318+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:28:19.442 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:28:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:28:19.318+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:28:19.443 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:28:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:28:19.319+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:28:19.443 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:28:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:28:19.110+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:28:19.443 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:28:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:28:19.110+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:28:19.443 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:28:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:28:19.110+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:28:19.443 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:28:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:28:19.110+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:28:19.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:28:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:28:19.441+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:28:19.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:28:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:28:19.441+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:28:19.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:28:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:28:19.441+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:28:19.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:28:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:28:19.442+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:28:20.431 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:28:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:28:20.188+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 4623 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:28:20.431 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:28:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:28:20.273+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:28:20.431 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:28:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:28:20.273+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:28:20.431 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:28:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:28:20.273+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:28:20.431 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:28:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:28:20.273+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:28:20.432 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:28:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:28:20.135+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:28:20.432 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:28:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:28:20.135+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:28:20.432 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:28:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:28:20.135+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:28:20.432 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:28:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:28:20.136+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:28:20.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:28:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:28:20.430+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:28:20.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:28:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:28:20.430+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:28:20.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:28:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:28:20.430+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:28:20.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:28:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:28:20.431+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:28:21.424 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:28:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:28:21.311+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:28:21.424 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:28:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:28:21.311+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:28:21.425 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:28:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:28:21.311+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:28:21.425 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:28:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:28:21.311+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:28:21.425 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:28:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:28:21.140+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:28:21.425 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:28:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:28:21.140+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:28:21.425 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:28:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:28:21.140+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:28:21.425 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:28:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:28:21.141+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:28:21.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:28:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:28:21.424+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:28:21.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:28:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:28:21.424+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:28:21.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:28:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:28:21.424+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:28:21.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:28:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:28:21.424+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:28:22.300 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:28:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:28:22.102+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:28:22.300 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:28:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:28:22.102+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:28:22.300 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:28:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:28:22.102+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:28:22.300 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:28:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:28:22.103+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:28:22.569 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:28:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:28:22.299+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:28:22.569 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:28:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:28:22.299+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:28:22.569 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:28:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:28:22.299+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:28:22.569 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:28:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:28:22.299+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:28:22.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:28:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:28:22.449+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:28:22.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:28:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:28:22.449+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:28:22.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:28:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:28:22.449+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:28:22.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:28:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:28:22.450+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:28:23.483 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:28:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:28:23.132+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:28:23.483 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:28:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:28:23.132+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:28:23.483 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:28:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:28:23.132+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:28:23.483 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:28:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:28:23.132+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:28:23.483 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:28:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:28:23.267+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:28:23.483 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:28:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:28:23.267+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:28:23.484 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:28:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:28:23.267+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:28:23.484 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:28:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:28:23.268+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:28:23.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:28:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:28:23.482+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:28:23.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:28:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:28:23.482+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:28:23.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:28:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:28:23.482+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:28:23.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:28:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:28:23.483+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:28:24.463 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:28:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:28:24.251+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:28:24.463 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:28:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:28:24.251+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:28:24.463 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:28:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:28:24.251+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:28:24.463 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:28:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:28:24.252+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:28:24.463 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:28:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:28:24.130+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:28:24.463 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:28:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:28:24.130+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:28:24.463 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:28:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:28:24.130+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:28:24.463 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:28:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:28:24.131+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:28:24.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:28:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:28:24.462+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:28:24.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:28:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:28:24.462+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:28:24.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:28:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:28:24.462+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:28:24.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:28:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:28:24.462+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:28:25.513 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:28:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:28:25.189+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 4717 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:28:25.513 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:28:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:28:25.283+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:28:25.513 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:28:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:28:25.283+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:28:25.513 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:28:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:28:25.283+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:28:25.513 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:28:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:28:25.284+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:28:25.513 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:28:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:28:25.112+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:28:25.514 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:28:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:28:25.112+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:28:25.514 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:28:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:28:25.112+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:28:25.514 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:28:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:28:25.112+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:28:25.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:28:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:28:25.512+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:28:25.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:28:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:28:25.512+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:28:25.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:28:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:28:25.512+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:28:25.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:28:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:28:25.513+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:28:26.481 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:28:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:28:26.298+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:28:26.481 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:28:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:28:26.298+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:28:26.481 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:28:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:28:26.298+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:28:26.481 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:28:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:28:26.299+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:28:26.481 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:28:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:28:26.153+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:28:26.481 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:28:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:28:26.153+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:28:26.481 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:28:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:28:26.153+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:28:26.481 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:28:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:28:26.154+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:28:26.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:28:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:28:26.480+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:28:26.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:28:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:28:26.480+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:28:26.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:28:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:28:26.480+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:28:26.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:28:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:28:26.480+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:28:27.512 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:28:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:28:27.173+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:28:27.512 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:28:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:28:27.173+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:28:27.512 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:28:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:28:27.173+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:28:27.512 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:28:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:28:27.174+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:28:27.512 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:28:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:28:27.260+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:28:27.512 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:28:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:28:27.260+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:28:27.512 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:28:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:28:27.260+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:28:27.512 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:28:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:28:27.261+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:28:27.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:28:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:28:27.511+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:28:27.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:28:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:28:27.511+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:28:27.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:28:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:28:27.511+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:28:27.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:28:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:28:27.511+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:28:28.543 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:28:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:28:28.186+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:28:28.543 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:28:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:28:28.186+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:28:28.543 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:28:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:28:28.186+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:28:28.543 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:28:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:28:28.187+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:28:28.543 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:28:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:28:28.270+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:28:28.543 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:28:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:28:28.270+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:28:28.544 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:28:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:28:28.270+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:28:28.544 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:28:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:28:28.271+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:28:28.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:28:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:28:28.542+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:28:28.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:28:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:28:28.542+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:28:28.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:28:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:28:28.542+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:28:28.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:28:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:28:28.543+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:28:29.512 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:28:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:28:29.186+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:28:29.512 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:28:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:28:29.186+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:28:29.512 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:28:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:28:29.186+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:28:29.512 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:28:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:28:29.187+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:28:29.512 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:28:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:28:29.285+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:28:29.512 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:28:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:28:29.285+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:28:29.512 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:28:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:28:29.285+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:28:29.512 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:28:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:28:29.286+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:28:29.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:28:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:28:29.510+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:28:29.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:28:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:28:29.510+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:28:29.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:28:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:28:29.510+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:28:29.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:28:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:28:29.511+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:28:30.551 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:28:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:28:30.190+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 4822 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:28:30.552 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:28:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:28:30.319+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:28:30.552 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:28:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:28:30.319+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:28:30.552 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:28:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:28:30.319+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:28:30.552 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:28:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:28:30.320+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:28:30.552 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:28:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:28:30.203+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:28:30.552 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:28:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:28:30.203+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:28:30.552 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:28:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:28:30.203+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:28:30.552 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:28:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:28:30.204+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:28:30.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:28:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:28:30.551+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:28:30.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:28:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:28:30.551+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:28:30.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:28:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:28:30.551+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:28:30.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:28:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:28:30.551+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:28:31.569 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:28:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:28:31.221+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:28:31.569 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:28:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:28:31.221+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:28:31.569 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:28:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:28:31.221+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:28:31.569 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:28:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:28:31.221+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:28:31.569 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:28:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:28:31.301+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:28:31.570 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:28:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:28:31.301+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:28:31.570 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:28:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:28:31.301+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:28:31.570 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:28:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:28:31.301+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:28:32.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:28:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:28:31.576+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:28:32.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:28:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:28:31.576+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:28:32.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:28:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:28:31.576+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:28:32.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:28:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:28:31.577+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:28:32.532 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:28:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:28:32.237+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:28:32.532 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:28:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:28:32.237+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:28:32.532 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:28:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:28:32.237+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:28:32.532 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:28:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:28:32.237+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:28:32.532 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:28:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:28:32.289+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:28:32.532 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:28:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:28:32.289+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:28:32.532 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:28:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:28:32.289+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:28:32.532 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:28:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:28:32.290+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:28:32.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:28:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:28:32.531+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:28:32.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:28:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:28:32.531+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:28:32.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:28:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:28:32.531+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:28:32.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:28:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:28:32.532+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:28:33.526 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:28:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:28:33.189+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:28:33.526 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:28:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:28:33.189+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:28:33.526 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:28:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:28:33.189+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:28:33.526 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:28:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:28:33.190+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:28:33.527 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:28:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:28:33.339+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:28:33.527 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:28:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:28:33.339+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:28:33.527 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:28:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:28:33.339+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:28:33.527 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:28:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:28:33.340+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:28:33.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:28:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:28:33.525+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:28:33.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:28:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:28:33.525+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:28:33.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:28:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:28:33.525+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:28:33.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:28:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:28:33.526+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:28:34.569 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:28:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:28:34.165+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:28:34.569 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:28:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:28:34.165+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:28:34.569 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:28:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:28:34.165+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:28:34.569 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:28:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:28:34.166+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:28:34.569 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:28:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:28:34.318+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:28:34.570 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:28:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:28:34.318+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:28:34.570 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:28:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:28:34.318+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:28:34.570 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:28:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:28:34.319+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:28:35.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:28:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:28:34.572+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:28:35.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:28:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:28:34.572+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:28:35.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:28:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:28:34.572+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:28:35.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:28:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:28:34.572+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:28:35.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:28:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:28:35.191+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 4914 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:28:35.570 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:28:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:28:35.344+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:28:35.570 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:28:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:28:35.344+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:28:35.570 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:28:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:28:35.344+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:28:35.570 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:28:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:28:35.345+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:28:35.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:28:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:28:35.130+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:28:35.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:28:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:28:35.130+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:28:35.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:28:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:28:35.130+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:28:35.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:28:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:28:35.130+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:28:36.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:28:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:28:35.599+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:28:36.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:28:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:28:35.599+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:28:36.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:28:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:28:35.599+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:28:36.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:28:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:28:35.600+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:28:36.563 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:28:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:28:36.090+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:28:36.563 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:28:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:28:36.090+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:28:36.563 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:28:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:28:36.090+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:28:36.563 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:28:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:28:36.091+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:28:36.564 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:28:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:28:36.306+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:28:36.564 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:28:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:28:36.306+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:28:36.564 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:28:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:28:36.306+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:28:36.564 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:28:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:28:36.307+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:28:36.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:28:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:28:36.562+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:28:36.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:28:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:28:36.562+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:28:36.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:28:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:28:36.562+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:28:36.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:28:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:28:36.562+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:28:37.545 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:28:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:28:37.354+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:28:37.545 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:28:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:28:37.354+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:28:37.545 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:28:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:28:37.354+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:28:37.545 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:28:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:28:37.355+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:28:37.546 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:28:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:28:37.110+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:28:37.546 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:28:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:28:37.110+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:28:37.546 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:28:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:28:37.110+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:28:37.546 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:28:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:28:37.110+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:28:37.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:28:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:28:37.544+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:28:37.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:28:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:28:37.544+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:28:37.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:28:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:28:37.544+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:28:37.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:28:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:28:37.545+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:28:38.375 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:28:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:28:38.090+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:28:38.375 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:28:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:28:38.090+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:28:38.376 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:28:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:28:38.090+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:28:38.376 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:28:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:28:38.091+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:28:38.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:28:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:28:38.375+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:28:38.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:28:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:28:38.375+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:28:38.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:28:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:28:38.375+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:28:38.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:28:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:28:38.376+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:28:38.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:28:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:28:38.587+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:28:38.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:28:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:28:38.587+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:28:38.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:28:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:28:38.587+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:28:38.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:28:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:28:38.587+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:28:39.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:28:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:28:39.066+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:28:39.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:28:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:28:39.066+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:28:39.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:28:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:28:39.066+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:28:39.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:28:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:28:39.066+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:28:39.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:28:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:28:39.573+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:28:39.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:28:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:28:39.573+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:28:39.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:28:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:28:39.573+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:28:39.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:28:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:28:39.574+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:28:39.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:28:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:28:39.379+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:28:39.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:28:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:28:39.379+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:28:39.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:28:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:28:39.379+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:28:39.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:28:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:28:39.380+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:28:40.317 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:28:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:28:40.192+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 5008 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:28:40.317 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:28:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:28:40.047+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:28:40.317 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:28:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:28:40.047+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:28:40.317 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:28:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:28:40.047+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:28:40.317 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:28:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:28:40.048+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:28:40.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:28:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:28:40.559+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:28:40.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:28:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:28:40.559+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:28:40.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:28:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:28:40.559+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:28:40.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:28:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:28:40.560+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:28:40.569 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:28:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:28:40.409+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:28:40.570 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:28:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:28:40.409+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:28:40.570 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:28:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:28:40.409+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:28:40.570 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:28:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:28:40.410+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:28:41.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:28:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:28:41.028+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:28:41.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:28:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:28:41.028+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:28:41.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:28:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:28:41.028+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:28:41.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:28:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:28:41.028+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:28:41.655 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:28:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:28:41.530+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:28:41.655 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:28:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:28:41.530+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:28:41.655 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:28:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:28:41.530+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:28:41.655 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:28:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:28:41.531+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:28:41.655 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:28:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:28:41.365+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:28:41.655 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:28:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:28:41.365+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:28:41.656 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:28:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:28:41.365+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:28:41.656 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:28:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:28:41.365+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:28:42.322 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:28:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:28:42.059+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:28:42.322 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:28:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:28:42.059+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:28:42.322 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:28:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:28:42.059+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:28:42.322 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:28:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:28:42.060+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:28:42.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:28:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:28:42.335+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:28:42.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:28:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:28:42.335+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:28:42.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:28:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:28:42.335+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:28:42.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:28:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:28:42.336+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:28:42.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:28:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:28:42.523+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:28:42.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:28:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:28:42.523+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:28:42.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:28:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:28:42.523+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:28:42.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:28:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:28:42.523+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:28:43.542 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:28:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:28:43.100+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:28:43.542 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:28:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:28:43.100+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:28:43.542 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:28:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:28:43.100+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:28:43.542 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:28:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:28:43.100+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:28:43.543 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:28:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:28:43.348+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:28:43.543 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:28:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:28:43.348+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:28:43.543 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:28:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:28:43.348+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:28:43.543 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:28:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:28:43.348+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:28:43.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:28:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:28:43.569+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:28:43.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:28:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:28:43.569+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:28:43.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:28:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:28:43.569+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:28:43.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:28:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:28:43.570+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:28:44.343 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:28:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:28:44.075+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:28:44.343 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:28:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:28:44.075+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:28:44.343 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:28:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:28:44.075+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:28:44.343 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:28:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:28:44.075+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:28:44.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:28:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:28:44.342+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:28:44.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:28:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:28:44.342+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:28:44.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:28:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:28:44.342+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:28:44.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:28:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:28:44.344+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:28:44.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:28:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:28:44.579+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:28:44.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:28:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:28:44.579+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:28:44.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:28:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:28:44.579+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:28:44.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:28:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:28:44.580+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:28:45.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:28:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:28:45.193+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 5120 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:28:45.570 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:28:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:28:45.345+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:28:45.570 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:28:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:28:45.345+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:28:45.570 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:28:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:28:45.345+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:28:45.570 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:28:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:28:45.346+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:28:45.571 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:28:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:28:45.113+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:28:45.571 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:28:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:28:45.113+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:28:45.571 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:28:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:28:45.113+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:28:45.571 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:28:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:28:45.113+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:28:46.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:28:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:28:45.576+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:28:46.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:28:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:28:45.576+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:28:46.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:28:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:28:45.576+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:28:46.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:28:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:28:45.577+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:28:46.385 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:28:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:28:46.103+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:28:46.385 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:28:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:28:46.103+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:28:46.385 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:28:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:28:46.103+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:28:46.385 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:28:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:28:46.103+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:28:46.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:28:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:28:46.384+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:28:46.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:28:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:28:46.384+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:28:46.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:28:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:28:46.384+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:28:46.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:28:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:28:46.385+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:28:46.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:28:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:28:46.592+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:28:46.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:28:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:28:46.592+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:28:46.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:28:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:28:46.592+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:28:46.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:28:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:28:46.593+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:28:47.414 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:28:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:28:47.090+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:28:47.415 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:28:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:28:47.090+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:28:47.415 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:28:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:28:47.090+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:28:47.415 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:28:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:28:47.091+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:28:47.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:28:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:28:47.555+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:28:47.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:28:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:28:47.555+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:28:47.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:28:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:28:47.555+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:28:47.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:28:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:28:47.555+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:28:47.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:28:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:28:47.414+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:28:47.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:28:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:28:47.414+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:28:47.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:28:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:28:47.414+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:28:47.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:28:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:28:47.415+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:28:48.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:28:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:28:48.046+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:28:48.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:28:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:28:48.046+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:28:48.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:28:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:28:48.046+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:28:48.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:28:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:28:48.047+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:28:48.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:28:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:28:48.394+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:28:48.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:28:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:28:48.394+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:28:48.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:28:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:28:48.394+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:28:48.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:28:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:28:48.395+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:28:48.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:28:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:28:48.528+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:28:48.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:28:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:28:48.528+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:28:48.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:28:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:28:48.528+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:28:48.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:28:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:28:48.529+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:28:49.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:28:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:28:49.042+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:28:49.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:28:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:28:49.042+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:28:49.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:28:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:28:49.042+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:28:49.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:28:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:28:49.043+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:28:49.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:28:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:28:49.400+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:28:49.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:28:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:28:49.400+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:28:49.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:28:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:28:49.400+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:28:49.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:28:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:28:49.401+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:28:49.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:28:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:28:49.506+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:28:49.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:28:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:28:49.506+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:28:49.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:28:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:28:49.506+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:28:49.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:28:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:28:49.507+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:28:50.317 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:28:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:28:50.194+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 5237 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:28:50.317 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:28:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:28:50.041+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:28:50.317 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:28:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:28:50.041+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:28:50.317 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:28:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:28:50.041+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:28:50.317 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:28:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:28:50.041+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:28:50.569 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:28:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:28:50.392+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:28:50.569 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:28:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:28:50.392+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:28:50.569 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:28:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:28:50.392+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:28:50.569 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:28:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:28:50.394+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:28:50.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:28:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:28:50.462+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:28:50.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:28:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:28:50.462+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:28:50.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:28:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:28:50.462+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:28:50.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:28:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:28:50.462+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:28:51.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:28:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:28:51.016+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:28:51.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:28:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:28:51.016+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:28:51.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:28:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:28:51.016+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:28:51.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:28:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:28:51.016+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:28:51.655 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:28:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:28:51.439+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:28:51.656 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:28:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:28:51.439+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:28:51.656 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:28:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:28:51.439+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:28:51.656 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:28:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:28:51.440+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:28:51.656 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:28:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:28:51.349+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:28:51.656 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:28:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:28:51.349+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:28:51.656 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:28:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:28:51.349+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:28:51.656 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:28:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:28:51.350+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:28:52.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:28:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:28:51.972+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:28:52.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:28:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:28:51.972+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:28:52.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:28:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:28:51.972+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:28:52.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:28:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:28:51.973+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:28:52.569 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:28:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:28:52.317+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:28:52.569 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:28:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:28:52.317+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:28:52.569 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:28:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:28:52.317+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:28:52.569 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:28:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:28:52.318+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:28:52.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:28:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:28:52.445+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:28:52.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:28:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:28:52.445+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:28:52.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:28:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:28:52.445+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:28:52.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:28:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:28:52.446+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:28:53.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:28:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:28:52.967+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:28:53.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:28:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:28:52.967+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:28:53.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:28:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:28:52.967+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:28:53.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:28:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:28:52.967+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:28:53.569 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:28:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:28:53.315+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:28:53.570 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:28:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:28:53.315+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:28:53.570 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:28:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:28:53.315+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:28:53.570 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:28:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:28:53.316+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:28:53.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:28:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:28:53.429+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:28:53.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:28:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:28:53.429+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:28:53.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:28:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:28:53.429+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:28:53.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:28:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:28:53.429+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:28:54.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:28:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:28:53.934+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:28:54.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:28:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:28:53.934+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:28:54.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:28:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:28:53.934+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:28:54.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:28:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:28:53.935+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:28:54.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:28:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:28:54.407+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:28:54.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:28:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:28:54.407+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:28:54.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:28:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:28:54.407+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:28:54.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:28:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:28:54.407+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:28:54.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:28:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:28:54.343+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:28:54.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:28:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:28:54.343+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:28:54.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:28:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:28:54.343+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:28:54.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:28:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:28:54.343+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:28:55.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:28:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:28:54.890+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:28:55.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:28:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:28:54.890+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:28:55.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:28:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:28:54.890+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:28:55.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:28:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:28:54.890+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:28:55.452 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:28:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:28:55.193+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 5328 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:28:55.452 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:28:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:28:55.391+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:28:55.452 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:28:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:28:55.391+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:28:55.452 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:28:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:28:55.391+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:28:55.452 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:28:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:28:55.391+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:28:55.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:28:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:28:55.450+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:28:55.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:28:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:28:55.450+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:28:55.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:28:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:28:55.450+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:28:55.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:28:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:28:55.451+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:28:56.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:28:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:28:55.922+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:28:56.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:28:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:28:55.922+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:28:56.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:28:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:28:55.922+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:28:56.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:28:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:28:55.922+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:28:56.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:28:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:28:56.433+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:28:56.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:28:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:28:56.433+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:28:56.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:28:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:28:56.433+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:28:56.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:28:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:28:56.433+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:28:56.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:28:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:28:56.445+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:28:56.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:28:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:28:56.445+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:28:56.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:28:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:28:56.445+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:28:56.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:28:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:28:56.446+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:28:57.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:28:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:28:56.894+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:28:57.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:28:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:28:56.894+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:28:57.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:28:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:28:56.894+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:28:57.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:28:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:28:56.895+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:28:57.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:28:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:28:57.469+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:28:57.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:28:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:28:57.469+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:28:57.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:28:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:28:57.469+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:28:57.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:28:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:28:57.470+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:28:57.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:28:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:28:57.425+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:28:57.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:28:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:28:57.425+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:28:57.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:28:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:28:57.425+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:28:57.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:28:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:28:57.426+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:28:58.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:28:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:28:57.921+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:28:58.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:28:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:28:57.921+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:28:58.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:28:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:28:57.921+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:28:58.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:28:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:28:57.922+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:28:58.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:28:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:28:58.514+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:28:58.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:28:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:28:58.514+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:28:58.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:28:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:28:58.514+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:28:58.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:28:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:28:58.515+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:28:58.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:28:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:28:58.443+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:28:58.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:28:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:28:58.443+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:28:58.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:28:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:28:58.443+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:28:58.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:28:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:28:58.443+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:28:59.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:28:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:28:58.935+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:28:59.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:28:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:28:58.935+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:28:59.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:28:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:28:58.935+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:28:59.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:28:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:28:58.935+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:28:59.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:28:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:28:59.453+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:28:59.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:28:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:28:59.453+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:28:59.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:28:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:28:59.453+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:28:59.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:28:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:28:59.454+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:28:59.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:28:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:28:59.485+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:28:59.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:28:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:28:59.486+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:28:59.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:28:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:28:59.486+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:28:59.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:28:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:28:59.486+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:29:00.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:28:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:28:59.903+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:29:00.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:28:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:28:59.903+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:29:00.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:28:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:28:59.903+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:29:00.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:28:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:28:59.903+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:29:00.526 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:29:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:29:00.194+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 5421 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:29:00.797 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:29:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:29:00.405+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:29:00.797 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:29:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:29:00.405+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:29:00.797 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:29:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:29:00.405+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:29:00.797 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:29:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:29:00.406+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:29:00.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:29:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:29:00.524+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:29:00.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:29:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:29:00.524+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:29:00.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:29:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:29:00.524+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:29:00.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:29:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:29:00.524+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:29:01.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:29:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:29:00.863+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:29:01.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:29:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:29:00.863+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:29:01.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:29:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:29:00.863+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:29:01.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:29:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:29:00.863+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:29:01.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:29:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:29:01.547+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:29:01.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:29:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:29:01.547+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:29:01.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:29:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:29:01.547+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:29:01.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:29:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:29:01.548+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:29:01.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:29:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:29:01.419+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:29:01.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:29:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:29:01.419+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:29:01.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:29:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:29:01.419+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:29:01.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:29:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:29:01.419+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:29:02.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:29:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:29:01.874+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:29:02.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:29:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:29:01.874+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:29:02.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:29:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:29:01.874+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:29:02.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:29:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:29:01.874+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:29:02.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:29:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:29:02.575+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:29:02.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:29:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:29:02.575+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:29:02.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:29:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:29:02.575+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:29:02.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:29:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:29:02.576+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:29:02.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:29:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:29:02.405+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:29:02.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:29:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:29:02.405+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:29:02.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:29:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:29:02.405+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:29:02.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:29:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:29:02.406+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:29:03.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:29:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:29:02.857+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:29:03.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:29:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:29:02.857+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:29:03.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:29:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:29:02.857+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:29:03.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:29:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:29:02.857+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:29:03.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:29:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:29:03.429+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:29:03.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:29:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:29:03.429+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:29:03.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:29:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:29:03.429+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:29:03.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:29:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:29:03.430+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:29:03.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:29:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:29:03.606+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:29:03.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:29:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:29:03.606+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:29:03.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:29:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:29:03.606+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:29:03.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:29:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:29:03.607+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:29:04.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:29:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:29:03.867+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:29:04.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:29:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:29:03.867+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:29:04.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:29:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:29:03.867+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:29:04.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:29:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:29:03.867+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:29:04.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:29:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:29:04.429+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:29:04.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:29:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:29:04.429+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:29:04.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:29:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:29:04.429+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:29:04.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:29:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:29:04.429+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:29:04.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:29:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:29:04.594+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:29:04.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:29:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:29:04.594+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:29:04.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:29:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:29:04.594+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:29:04.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:29:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:29:04.595+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:29:05.197 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:29:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:29:04.834+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:29:05.197 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:29:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:29:04.834+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:29:05.197 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:29:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:29:04.834+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:29:05.197 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:29:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:29:04.835+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:29:05.459 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:29:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:29:05.196+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 5515 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:29:05.813 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:29:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:29:05.459+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:29:05.813 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:29:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:29:05.459+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:29:05.814 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:29:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:29:05.459+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:29:05.814 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:29:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:29:05.459+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:29:05.814 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:29:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:29:05.594+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:29:05.814 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:29:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:29:05.594+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:29:05.814 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:29:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:29:05.594+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:29:05.814 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:29:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:29:05.595+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:29:06.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:29:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:29:05.813+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:29:06.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:29:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:29:05.813+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:29:06.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:29:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:29:05.813+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:29:06.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:29:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:29:05.813+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:29:06.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:29:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:29:06.574+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:29:06.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:29:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:29:06.574+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:29:06.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:29:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:29:06.574+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:29:06.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:29:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:29:06.575+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:29:06.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:29:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:29:06.495+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:29:06.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:29:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:29:06.495+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:29:06.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:29:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:29:06.495+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:29:06.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:29:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:29:06.496+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:29:07.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:29:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:29:06.849+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:29:07.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:29:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:29:06.849+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:29:07.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:29:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:29:06.849+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:29:07.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:29:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:29:06.849+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:29:07.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:29:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:29:07.578+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:29:07.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:29:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:29:07.578+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:29:07.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:29:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:29:07.578+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:29:07.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:29:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:29:07.579+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:29:07.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:29:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:29:07.461+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:29:07.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:29:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:29:07.461+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:29:07.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:29:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:29:07.461+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:29:07.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:29:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:29:07.461+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:29:08.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:29:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:29:07.880+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:29:08.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:29:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:29:07.880+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:29:08.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:29:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:29:07.880+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:29:08.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:29:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:29:07.881+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:29:08.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:29:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:29:08.482+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:29:08.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:29:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:29:08.482+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:29:08.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:29:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:29:08.482+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:29:08.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:29:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:29:08.483+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:29:08.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:29:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:29:08.557+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:29:08.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:29:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:29:08.557+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:29:08.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:29:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:29:08.557+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:29:08.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:29:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:29:08.558+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:29:09.115 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:29:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:29:08.847+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:29:09.116 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:29:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:29:08.847+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:29:09.116 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:29:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:29:08.847+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:29:09.116 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:29:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:29:08.847+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:29:09.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:29:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:29:09.546+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:29:09.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:29:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:29:09.546+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:29:09.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:29:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:29:09.546+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:29:09.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:29:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:29:09.547+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:29:09.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:29:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:29:09.480+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:29:09.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:29:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:29:09.480+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:29:09.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:29:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:29:09.480+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:29:09.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:29:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:29:09.481+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:29:10.198 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:29:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:29:09.850+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:29:10.198 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:29:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:29:09.850+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:29:10.198 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:29:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:29:09.850+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:29:10.198 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:29:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:29:09.850+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:29:10.482 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:29:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:29:10.197+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 5607 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:29:10.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:29:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:29:10.595+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:29:10.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:29:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:29:10.595+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:29:10.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:29:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:29:10.595+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:29:10.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:29:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:29:10.596+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:29:10.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:29:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:29:10.481+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:29:10.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:29:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:29:10.481+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:29:10.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:29:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:29:10.481+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:29:10.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:29:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:29:10.482+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:29:11.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:29:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:29:10.854+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:29:11.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:29:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:29:10.854+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:29:11.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:29:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:29:10.854+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:29:11.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:29:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:29:10.854+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:29:11.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:29:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:29:11.492+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:29:11.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:29:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:29:11.492+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:29:11.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:29:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:29:11.492+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:29:11.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:29:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:29:11.493+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:29:11.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:29:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:29:11.598+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:29:11.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:29:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:29:11.598+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:29:11.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:29:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:29:11.598+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:29:11.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:29:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:29:11.599+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:29:12.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:29:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:29:11.857+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:29:12.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:29:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:29:11.857+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:29:12.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:29:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:29:11.858+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:29:12.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:29:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:29:11.858+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:29:12.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:29:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:29:12.480+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:29:12.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:29:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:29:12.480+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:29:12.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:29:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:29:12.480+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:29:12.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:29:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:29:12.481+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:29:12.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:29:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:29:12.628+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:29:12.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:29:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:29:12.628+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:29:12.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:29:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:29:12.628+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:29:12.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:29:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:29:12.629+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:29:13.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:29:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:29:12.843+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:29:13.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:29:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:29:12.843+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:29:13.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:29:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:29:12.843+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:29:13.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:29:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:29:12.843+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:29:13.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:29:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:29:13.604+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:29:13.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:29:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:29:13.604+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:29:13.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:29:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:29:13.604+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:29:13.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:29:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:29:13.605+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:29:13.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:29:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:29:13.433+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:29:13.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:29:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:29:13.433+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:29:13.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:29:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:29:13.433+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:29:13.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:29:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:29:13.434+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:29:14.267 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:29:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:29:13.888+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:29:14.267 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:29:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:29:13.888+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:29:14.267 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:29:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:29:13.888+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:29:14.267 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:29:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:29:13.889+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:29:14.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:29:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:29:14.462+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:29:14.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:29:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:29:14.462+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:29:14.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:29:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:29:14.462+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:29:14.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:29:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:29:14.463+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:29:14.905 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:29:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:29:14.613+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:29:14.905 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:29:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:29:14.613+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:29:14.905 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:29:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:29:14.613+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:29:14.905 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:29:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:29:14.614+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:29:15.198 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:29:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:29:14.904+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:29:15.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:29:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:29:14.904+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:29:15.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:29:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:29:14.904+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:29:15.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:29:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:29:14.905+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:29:15.459 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:29:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:29:15.198+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 5699 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:29:15.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:29:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:29:15.459+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:29:15.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:29:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:29:15.459+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:29:15.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:29:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:29:15.459+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:29:15.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:29:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:29:15.460+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:29:15.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:29:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:29:15.623+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:29:15.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:29:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:29:15.623+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:29:15.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:29:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:29:15.623+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:29:15.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:29:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:29:15.624+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:29:16.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:29:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:29:15.908+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:29:16.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:29:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:29:15.908+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:29:16.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:29:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:29:15.908+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:29:16.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:29:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:29:15.909+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:29:16.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:29:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:29:16.641+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:29:16.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:29:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:29:16.641+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:29:16.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:29:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:29:16.641+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:29:16.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:29:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:29:16.641+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:29:16.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:29:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:29:16.498+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:29:16.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:29:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:29:16.498+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:29:16.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:29:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:29:16.498+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:29:16.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:29:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:29:16.498+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:29:17.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:29:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:29:16.951+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:29:17.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:29:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:29:16.951+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:29:17.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:29:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:29:16.951+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:29:17.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:29:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:29:16.952+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:29:17.677 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:29:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:29:17.528+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:29:17.677 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:29:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:29:17.528+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:29:17.677 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:29:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:29:17.528+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:29:17.677 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:29:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:29:17.529+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:29:18.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:29:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:29:17.676+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:29:18.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:29:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:29:17.676+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:29:18.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:29:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:29:17.676+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:29:18.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:29:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:29:17.677+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:29:18.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:29:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:29:17.924+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:29:18.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:29:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:29:17.924+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:29:18.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:29:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:29:17.924+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:29:18.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:29:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:29:17.925+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:29:18.892 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:29:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:29:18.661+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:29:18.893 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:29:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:29:18.661+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:29:18.893 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:29:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:29:18.661+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:29:18.893 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:29:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:29:18.662+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:29:18.893 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:29:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:29:18.575+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:29:18.893 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:29:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:29:18.575+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:29:18.893 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:29:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:29:18.576+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:29:18.893 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:29:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:29:18.576+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:29:19.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:29:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:29:18.891+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:29:19.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:29:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:29:18.892+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:29:19.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:29:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:29:18.892+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:29:19.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:29:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:29:18.892+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:29:19.896 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:29:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:29:19.584+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:29:19.896 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:29:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:29:19.584+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:29:19.896 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:29:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:29:19.584+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:29:19.896 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:29:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:29:19.584+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:29:19.896 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:29:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:29:19.686+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:29:19.896 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:29:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:29:19.686+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:29:19.896 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:29:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:29:19.686+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:29:19.896 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:29:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:29:19.687+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:29:20.200 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:29:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:29:19.895+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:29:20.200 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:29:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:29:19.895+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:29:20.200 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:29:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:29:19.895+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:29:20.200 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:29:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:29:19.895+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:29:20.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:29:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:29:20.199+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 5791 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:29:20.877 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:29:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:29:20.599+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:29:20.877 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:29:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:29:20.599+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:29:20.878 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:29:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:29:20.599+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:29:20.878 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:29:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:29:20.600+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:29:20.878 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:29:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:29:20.693+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:29:20.878 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:29:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:29:20.693+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:29:20.878 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:29:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:29:20.693+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:29:20.878 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:29:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:29:20.694+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:29:21.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:29:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:29:20.876+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:29:21.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:29:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:29:20.876+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:29:21.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:29:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:29:20.876+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:29:21.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:29:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:29:20.877+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:29:22.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:29:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:29:21.743+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:29:22.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:29:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:29:21.743+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:29:22.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:29:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:29:21.743+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:29:22.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:29:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:29:21.744+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:29:22.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:29:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:29:21.894+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:29:22.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:29:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:29:21.894+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:29:22.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:29:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:29:21.894+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:29:22.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:29:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:29:21.894+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:29:22.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:29:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:29:21.650+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:29:22.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:29:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:29:21.650+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:29:22.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:29:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:29:21.650+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:29:22.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:29:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:29:21.650+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:29:23.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:29:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:29:22.852+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:29:23.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:29:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:29:22.852+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:29:23.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:29:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:29:22.852+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:29:23.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:29:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:29:22.853+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:29:23.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:29:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:29:22.706+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:29:23.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:29:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:29:22.706+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:29:23.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:29:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:29:22.706+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:29:23.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:29:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:29:22.707+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:29:23.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:29:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:29:22.643+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:29:23.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:29:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:29:22.643+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:29:23.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:29:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:29:22.643+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:29:23.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:29:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:29:22.644+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:29:23.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:29:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:29:23.676+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:29:23.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:29:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:29:23.676+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:29:23.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:29:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:29:23.676+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:29:23.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:29:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:29:23.677+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:29:23.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:29:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:29:23.619+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:29:23.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:29:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:29:23.619+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:29:23.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:29:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:29:23.619+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:29:23.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:29:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:29:23.620+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:29:24.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:29:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:29:23.839+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:29:24.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:29:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:29:23.839+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:29:24.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:29:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:29:23.839+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:29:24.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:29:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:29:23.839+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:29:25.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:29:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:29:24.636+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:29:25.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:29:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:29:24.636+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:29:25.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:29:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:29:24.636+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:29:25.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:29:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:29:24.637+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:29:25.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:29:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:29:24.628+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:29:25.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:29:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:29:24.628+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:29:25.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:29:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:29:24.628+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:29:25.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:29:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:29:24.628+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:29:25.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:29:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:29:24.858+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:29:25.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:29:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:29:24.859+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:29:25.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:29:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:29:24.859+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:29:25.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:29:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:29:24.859+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:29:25.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:29:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:29:25.200+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 5885 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:29:26.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:29:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:29:25.599+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:29:26.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:29:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:29:25.599+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:29:26.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:29:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:29:25.599+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:29:26.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:29:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:29:25.600+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:29:26.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:29:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:29:25.649+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:29:26.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:29:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:29:25.649+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:29:26.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:29:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:29:25.649+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:29:26.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:29:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:29:25.649+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:29:26.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:29:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:29:25.844+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:29:26.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:29:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:29:25.844+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:29:26.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:29:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:29:25.844+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:29:26.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:29:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:29:25.845+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:29:26.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:29:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:29:26.662+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:29:26.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:29:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:29:26.662+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:29:26.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:29:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:29:26.662+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:29:26.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:29:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:29:26.663+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:29:26.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:29:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:29:26.644+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:29:26.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:29:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:29:26.644+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:29:26.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:29:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:29:26.644+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:29:26.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:29:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:29:26.644+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:29:27.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:29:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:29:26.893+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:29:27.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:29:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:29:26.893+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:29:27.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:29:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:29:26.893+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:29:27.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:29:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:29:26.893+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:29:27.905 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:29:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:29:27.639+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:29:27.905 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:29:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:29:27.639+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:29:27.906 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:29:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:29:27.639+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:29:27.906 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:29:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:29:27.640+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:29:27.906 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:29:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:29:27.639+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:29:27.906 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:29:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:29:27.639+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:29:27.906 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:29:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:29:27.639+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:29:27.906 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:29:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:29:27.640+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:29:28.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:29:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:29:27.904+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:29:28.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:29:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:29:27.904+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:29:28.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:29:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:29:27.904+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:29:28.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:29:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:29:27.905+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:29:29.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:29:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:29:28.634+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:29:29.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:29:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:29:28.634+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:29:29.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:29:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:29:28.634+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:29:29.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:29:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:29:28.635+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:29:29.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:29:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:29:28.637+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:29:29.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:29:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:29:28.637+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:29:29.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:29:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:29:28.637+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:29:29.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:29:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:29:28.638+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:29:29.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:29:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:29:28.869+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:29:29.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:29:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:29:28.869+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:29:29.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:29:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:29:28.869+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:29:29.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:29:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:29:28.869+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:29:29.887 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:29:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:29:29.636+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:29:29.888 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:29:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:29:29.636+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:29:29.888 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:29:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:29:29.636+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:29:29.888 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:29:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:29:29.637+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:29:29.888 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:29:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:29:29.677+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:29:29.888 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:29:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:29:29.677+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:29:29.888 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:29:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:29:29.677+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:29:29.888 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:29:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:29:29.678+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:29:30.201 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:29:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:29:29.886+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:29:30.202 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:29:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:29:29.886+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:29:30.202 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:29:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:29:29.886+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:29:30.202 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:29:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:29:29.887+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:29:30.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:29:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:29:30.201+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 5979 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:29:31.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:29:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:29:30.641+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:29:31.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:29:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:29:30.641+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:29:31.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:29:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:29:30.641+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:29:31.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:29:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:29:30.642+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:29:31.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:29:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:29:30.684+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:29:31.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:29:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:29:30.684+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:29:31.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:29:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:29:30.684+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:29:31.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:29:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:29:30.684+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:29:31.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:29:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:29:30.837+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:29:31.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:29:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:29:30.837+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:29:31.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:29:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:29:30.837+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:29:31.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:29:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:29:30.838+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:29:32.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:29:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:29:31.681+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:29:32.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:29:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:29:31.681+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:29:32.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:29:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:29:31.681+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:29:32.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:29:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:29:31.682+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:29:32.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:29:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:29:31.669+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:29:32.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:29:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:29:31.669+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:29:32.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:29:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:29:31.669+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:29:32.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:29:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:29:31.670+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:29:32.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:29:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:29:31.823+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:29:32.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:29:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:29:31.823+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:29:32.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:29:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:29:31.823+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:29:32.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:29:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:29:31.824+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:29:33.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:29:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:29:32.675+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:29:33.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:29:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:29:32.675+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:29:33.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:29:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:29:32.675+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:29:33.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:29:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:29:32.676+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:29:33.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:29:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:29:32.652+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:29:33.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:29:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:29:32.652+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:29:33.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:29:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:29:32.652+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:29:33.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:29:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:29:32.653+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:29:33.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:29:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:29:32.776+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:29:33.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:29:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:29:32.776+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:29:33.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:29:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:29:32.776+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:29:33.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:29:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:29:32.776+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:29:33.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:29:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:29:33.649+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:29:33.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:29:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:29:33.649+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:29:33.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:29:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:29:33.649+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:29:33.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:29:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:29:33.650+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:29:33.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:29:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:29:33.657+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:29:33.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:29:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:29:33.657+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:29:33.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:29:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:29:33.657+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:29:33.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:29:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:29:33.658+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:29:34.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:29:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:29:33.821+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:29:34.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:29:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:29:33.821+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:29:34.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:29:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:29:33.821+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:29:34.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:29:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:29:33.821+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:29:35.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:29:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:29:34.636+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:29:35.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:29:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:29:34.636+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:29:35.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:29:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:29:34.636+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:29:35.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:29:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:29:34.637+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:29:35.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:29:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:29:34.681+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:29:35.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:29:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:29:34.681+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:29:35.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:29:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:29:34.681+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:29:35.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:29:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:29:34.683+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:29:35.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:29:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:29:34.824+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:29:35.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:29:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:29:34.824+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:29:35.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:29:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:29:34.824+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:29:35.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:29:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:29:34.825+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:29:35.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:29:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:29:35.202+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 6072 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:29:35.867 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:29:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:29:35.646+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:29:35.867 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:29:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:29:35.646+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:29:35.867 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:29:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:29:35.646+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:29:35.867 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:29:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:29:35.646+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:29:35.867 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:29:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:29:35.649+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:29:35.867 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:29:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:29:35.649+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:29:35.867 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:29:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:29:35.649+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:29:35.868 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:29:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:29:35.650+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:29:36.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:29:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:29:35.866+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:29:36.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:29:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:29:35.866+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:29:36.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:29:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:29:35.866+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:29:36.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:29:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:29:35.866+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:29:36.879 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:29:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:29:36.612+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:29:36.879 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:29:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:29:36.612+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:29:36.879 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:29:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:29:36.612+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:29:36.879 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:29:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:29:36.613+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:29:36.880 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:29:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:29:36.653+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:29:36.880 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:29:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:29:36.653+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:29:36.880 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:29:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:29:36.653+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:29:36.880 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:29:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:29:36.653+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:29:37.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:29:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:29:36.878+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:29:37.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:29:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:29:36.878+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:29:37.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:29:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:29:36.878+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:29:37.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:29:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:29:36.879+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:29:37.888 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:29:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:29:37.627+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:29:37.888 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:29:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:29:37.627+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:29:37.888 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:29:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:29:37.627+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:29:37.888 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:29:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:29:37.628+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:29:37.889 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:29:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:29:37.669+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:29:37.889 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:29:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:29:37.669+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:29:37.889 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:29:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:29:37.669+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:29:37.889 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:29:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:29:37.669+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:29:38.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:29:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:29:37.887+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:29:38.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:29:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:29:37.887+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:29:38.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:29:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:29:37.887+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:29:38.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:29:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:29:37.888+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:29:38.921 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:29:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:29:38.636+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:29:38.922 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:29:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:29:38.636+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:29:38.922 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:29:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:29:38.636+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:29:38.922 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:29:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:29:38.637+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:29:38.922 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:29:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:29:38.643+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:29:38.922 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:29:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:29:38.643+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:29:38.922 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:29:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:29:38.643+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:29:38.922 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:29:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:29:38.643+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:29:39.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:29:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:29:38.921+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:29:39.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:29:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:29:38.921+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:29:39.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:29:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:29:38.921+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:29:39.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:29:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:29:38.921+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:29:39.953 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:29:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:29:39.679+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:29:39.953 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:29:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:29:39.679+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:29:39.953 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:29:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:29:39.679+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:29:39.953 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:29:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:29:39.680+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:29:39.953 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:29:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:29:39.660+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:29:39.953 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:29:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:29:39.660+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:29:39.953 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:29:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:29:39.660+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:29:39.954 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:29:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:29:39.660+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:29:40.204 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:29:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:29:39.952+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:29:40.204 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:29:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:29:39.952+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:29:40.204 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:29:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:29:39.952+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:29:40.204 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:29:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:29:39.952+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:29:40.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:29:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:29:40.203+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 6165 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:29:40.915 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:29:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:29:40.650+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:29:40.915 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:29:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:29:40.650+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:29:40.915 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:29:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:29:40.650+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:29:40.915 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:29:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:29:40.651+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:29:40.916 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:29:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:29:40.665+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:29:40.916 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:29:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:29:40.665+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:29:40.916 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:29:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:29:40.665+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:29:40.916 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:29:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:29:40.666+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:29:41.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:29:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:29:40.914+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:29:41.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:29:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:29:40.914+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:29:41.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:29:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:29:40.914+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:29:41.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:29:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:29:40.915+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:29:41.886 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:29:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:29:41.650+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:29:41.886 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:29:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:29:41.650+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:29:41.886 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:29:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:29:41.650+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:29:41.886 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:29:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:29:41.651+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:29:41.887 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:29:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:29:41.626+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:29:41.887 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:29:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:29:41.626+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:29:41.887 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:29:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:29:41.626+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:29:41.887 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:29:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:29:41.626+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:29:42.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:29:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:29:41.885+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:29:42.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:29:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:29:41.885+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:29:42.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:29:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:29:41.885+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:29:42.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:29:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:29:41.886+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:29:43.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:29:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:29:42.630+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:29:43.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:29:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:29:42.630+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:29:43.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:29:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:29:42.630+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:29:43.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:29:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:29:42.630+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:29:43.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:29:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:29:42.644+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:29:43.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:29:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:29:42.644+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:29:43.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:29:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:29:42.644+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:29:43.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:29:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:29:42.645+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:29:43.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:29:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:29:42.839+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:29:43.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:29:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:29:42.839+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:29:43.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:29:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:29:42.839+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:29:43.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:29:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:29:42.840+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:29:43.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:29:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:29:43.613+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:29:43.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:29:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:29:43.613+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:29:43.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:29:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:29:43.613+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:29:43.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:29:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:29:43.613+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:29:43.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:29:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:29:43.692+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:29:43.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:29:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:29:43.692+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:29:43.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:29:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:29:43.692+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:29:43.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:29:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:29:43.693+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:29:44.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:29:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:29:43.832+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:29:44.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:29:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:29:43.832+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:29:44.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:29:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:29:43.832+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:29:44.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:29:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:29:43.833+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:29:44.837 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:29:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:29:44.720+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:29:44.837 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:29:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:29:44.720+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:29:44.838 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:29:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:29:44.721+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:29:44.838 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:29:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:29:44.721+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:29:44.838 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:29:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:29:44.585+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:29:44.838 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:29:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:29:44.585+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:29:44.838 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:29:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:29:44.585+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:29:44.838 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:29:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:29:44.586+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:29:45.205 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:29:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:29:44.836+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:29:45.205 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:29:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:29:44.836+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:29:45.205 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:29:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:29:44.836+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:29:45.205 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:29:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:29:44.837+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:29:45.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:29:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:29:45.204+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 6259 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:29:45.871 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:29:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:29:45.710+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:29:45.871 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:29:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:29:45.710+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:29:45.871 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:29:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:29:45.710+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:29:45.871 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:29:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:29:45.711+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:29:45.871 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:29:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:29:45.587+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:29:45.871 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:29:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:29:45.587+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:29:45.871 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:29:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:29:45.587+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:29:45.872 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:29:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:29:45.588+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:29:46.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:29:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:29:45.870+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:29:46.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:29:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:29:45.870+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:29:46.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:29:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:29:45.870+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:29:46.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:29:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:29:45.871+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:29:46.889 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:29:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:29:46.686+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:29:46.889 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:29:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:29:46.686+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:29:46.889 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:29:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:29:46.686+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:29:46.889 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:29:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:29:46.686+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:29:46.889 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:29:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:29:46.626+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:29:46.889 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:29:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:29:46.626+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:29:46.889 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:29:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:29:46.626+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:29:46.889 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:29:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:29:46.627+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:29:47.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:29:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:29:46.888+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:29:47.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:29:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:29:46.888+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:29:47.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:29:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:29:46.888+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:29:47.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:29:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:29:46.888+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:29:48.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:29:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:29:47.844+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:29:48.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:29:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:29:47.844+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:29:48.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:29:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:29:47.844+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:29:48.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:29:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:29:47.844+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:29:48.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:29:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:29:47.638+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:29:48.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:29:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:29:47.638+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:29:48.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:29:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:29:47.638+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:29:48.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:29:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:29:47.639+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:29:48.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:29:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:29:47.693+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:29:48.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:29:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:29:47.693+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:29:48.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:29:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:29:47.693+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:29:48.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:29:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:29:47.694+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:29:48.863 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:29:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:29:48.741+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:29:48.863 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:29:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:29:48.741+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:29:48.863 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:29:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:29:48.741+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:29:48.863 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:29:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:29:48.742+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:29:48.863 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:29:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:29:48.595+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:29:48.864 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:29:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:29:48.595+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:29:48.864 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:29:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:29:48.595+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:29:48.864 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:29:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:29:48.596+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:29:49.116 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:29:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:29:48.862+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:29:49.116 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:29:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:29:48.862+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:29:49.116 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:29:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:29:48.862+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:29:49.116 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:29:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:29:48.863+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:29:49.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:29:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:29:49.693+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:29:49.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:29:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:29:49.693+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:29:49.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:29:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:29:49.693+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:29:49.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:29:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:29:49.693+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:29:49.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:29:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:29:49.560+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:29:49.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:29:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:29:49.560+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:29:49.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:29:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:29:49.560+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:29:49.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:29:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:29:49.561+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:29:50.206 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:29:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:29:49.897+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:29:50.206 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:29:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:29:49.897+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:29:50.206 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:29:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:29:49.897+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:29:50.206 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:29:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:29:49.898+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:29:50.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:29:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:29:50.205+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 6352 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:29:50.882 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:29:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:29:50.604+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:29:50.882 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:29:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:29:50.604+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:29:50.882 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:29:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:29:50.604+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:29:50.883 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:29:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:29:50.605+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:29:50.883 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:29:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:29:50.720+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:29:50.883 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:29:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:29:50.720+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:29:50.883 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:29:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:29:50.720+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:29:50.883 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:29:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:29:50.721+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:29:51.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:29:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:29:50.881+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:29:51.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:29:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:29:50.881+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:29:51.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:29:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:29:50.881+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:29:51.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:29:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:29:50.882+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:29:52.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:29:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:29:51.618+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:29:52.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:29:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:29:51.618+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:29:52.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:29:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:29:51.618+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:29:52.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:29:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:29:51.619+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:29:52.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:29:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:29:51.750+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:29:52.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:29:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:29:51.750+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:29:52.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:29:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:29:51.750+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:29:52.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:29:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:29:51.750+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:29:52.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:29:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:29:51.858+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:29:52.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:29:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:29:51.858+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:29:52.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:29:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:29:51.858+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:29:52.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:29:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:29:51.859+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:29:53.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:29:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:29:52.834+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:29:53.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:29:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:29:52.834+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:29:53.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:29:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:29:52.834+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:29:53.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:29:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:29:52.834+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:29:53.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:29:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:29:52.701+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:29:53.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:29:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:29:52.701+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:29:53.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:29:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:29:52.701+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:29:53.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:29:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:29:52.702+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:29:53.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:29:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:29:52.623+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:29:53.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:29:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:29:52.623+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:29:53.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:29:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:29:52.623+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:29:53.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:29:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:29:52.624+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:29:53.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:29:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:29:53.578+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:29:53.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:29:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:29:53.578+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:29:53.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:29:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:29:53.578+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:29:53.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:29:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:29:53.578+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:29:53.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:29:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:29:53.708+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:29:53.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:29:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:29:53.708+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:29:53.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:29:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:29:53.708+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:29:53.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:29:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:29:53.709+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:29:53.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:29:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:29:53.788+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:29:53.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:29:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:29:53.788+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:29:53.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:29:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:29:53.788+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:29:53.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:29:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:29:53.788+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:29:54.802 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:29:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:29:54.547+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:29:54.802 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:29:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:29:54.547+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:29:54.802 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:29:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:29:54.547+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:29:54.802 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:29:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:29:54.548+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:29:54.802 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:29:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:29:54.694+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:29:54.802 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:29:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:29:54.694+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:29:54.802 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:29:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:29:54.694+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:29:54.802 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:29:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:29:54.694+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:29:55.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:29:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:29:54.800+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:29:55.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:29:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:29:54.800+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:29:55.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:29:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:29:54.800+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:29:55.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:29:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:29:54.800+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:29:55.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:29:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:29:55.205+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 6445 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:29:56.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:29:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:29:55.592+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:29:56.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:29:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:29:55.592+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:29:56.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:29:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:29:55.592+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:29:56.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:29:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:29:55.593+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:29:56.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:29:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:29:55.688+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:29:56.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:29:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:29:55.688+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:29:56.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:29:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:29:55.688+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:29:56.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:29:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:29:55.688+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:29:56.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:29:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:29:55.775+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:29:56.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:29:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:29:55.775+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:29:56.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:29:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:29:55.775+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:29:56.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:29:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:29:55.775+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:29:56.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:29:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:29:56.736+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:29:56.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:29:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:29:56.736+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:29:56.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:29:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:29:56.736+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:29:56.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:29:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:29:56.737+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:29:56.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:29:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:29:56.555+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:29:56.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:29:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:29:56.555+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:29:56.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:29:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:29:56.555+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:29:56.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:29:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:29:56.555+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:29:56.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:29:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:29:56.640+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:29:56.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:29:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:29:56.640+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:29:56.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:29:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:29:56.640+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:29:56.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:29:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:29:56.641+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:29:58.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:29:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:29:57.640+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:29:58.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:29:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:29:57.640+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:29:58.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:29:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:29:57.640+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:29:58.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:29:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:29:57.641+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:29:58.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:29:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:29:57.580+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:29:58.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:29:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:29:57.580+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:29:58.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:29:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:29:57.580+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:29:58.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:29:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:29:57.580+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:29:58.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:29:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:29:57.707+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:29:58.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:29:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:29:57.707+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:29:58.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:29:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:29:57.707+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:29:58.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:29:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:29:57.707+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:29:58.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:29:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:29:58.563+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:29:58.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:29:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:29:58.563+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:29:58.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:29:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:29:58.563+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:29:58.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:29:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:29:58.564+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:29:58.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:29:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:29:58.714+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:29:58.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:29:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:29:58.714+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:29:58.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:29:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:29:58.714+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:29:58.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:29:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:29:58.714+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:29:58.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:29:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:29:58.646+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:29:58.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:29:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:29:58.646+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:29:58.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:29:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:29:58.646+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:29:58.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:29:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:29:58.646+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:29:59.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:29:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:29:59.667+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:29:59.822 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:29:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:29:59.667+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:29:59.822 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:29:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:29:59.667+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:29:59.822 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:29:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:29:59.668+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:29:59.822 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:29:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:29:59.525+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:29:59.822 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:29:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:29:59.525+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:29:59.822 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:29:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:29:59.525+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:29:59.822 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:29:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:29:59.525+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:29:59.823 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:29:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:29:59.679+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:29:59.823 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:29:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:29:59.679+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:29:59.823 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:29:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:29:59.679+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:29:59.823 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:29:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:29:59.679+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:30:00.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:30:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:30:00.206+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 6537 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:30:01.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:30:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:30:00.569+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:30:01.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:30:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:30:00.569+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:30:01.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:30:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:30:00.569+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:30:01.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:30:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:30:00.570+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:30:01.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:30:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:30:00.725+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:30:01.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:30:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:30:00.725+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:30:01.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:30:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:30:00.725+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:30:01.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:30:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:30:00.725+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:30:01.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:30:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:30:00.690+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:30:01.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:30:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:30:00.690+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:30:01.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:30:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:30:00.690+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:30:01.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:30:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:30:00.690+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:30:02.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:30:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:30:01.592+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:30:02.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:30:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:30:01.592+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:30:02.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:30:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:30:01.592+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:30:02.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:30:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:30:01.593+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:30:02.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:30:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:30:01.689+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:30:02.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:30:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:30:01.689+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:30:02.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:30:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:30:01.689+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:30:02.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:30:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:30:01.689+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:30:02.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:30:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:30:01.689+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:30:02.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:30:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:30:01.689+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:30:02.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:30:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:30:01.689+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:30:02.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:30:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:30:01.689+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:30:03.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:30:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:30:02.596+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:30:03.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:30:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:30:02.596+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:30:03.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:30:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:30:02.596+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:30:03.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:30:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:30:02.597+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:30:03.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:30:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:30:02.738+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:30:03.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:30:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:30:02.738+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:30:03.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:30:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:30:02.738+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:30:03.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:30:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:30:02.739+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:30:03.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:30:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:30:02.675+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:30:03.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:30:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:30:02.675+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:30:03.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:30:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:30:02.675+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:30:03.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:30:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:30:02.675+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:30:03.672 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:30:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:30:03.560+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:30:03.673 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:30:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:30:03.560+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:30:03.673 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:30:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:30:03.560+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:30:03.673 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:30:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:30:03.561+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:30:04.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:30:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:30:03.672+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.926401+0000 front 2024-03-18T01:19:39.926488+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:30:04.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:30:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:30:03.672+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.926415+0000 front 2024-03-18T01:19:39.926470+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:30:04.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:30:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:30:03.672+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.926440+0000 front 2024-03-18T01:19:39.926529+0000 (oldest deadline 2024-03-18T01:20:04.026418+0000) 2024-03-18T01:30:04.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:30:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:30:03.672+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:30:04.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:30:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:30:03.698+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:30:04.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:30:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:30:03.698+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:30:04.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:30:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:30:03.698+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:30:04.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:30:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:30:03.698+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:30:04.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:30:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:30:04.714+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:30:04.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:30:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:30:04.524+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:30:04.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:30:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:30:04.524+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:30:04.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:30:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:30:04.524+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:30:04.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:30:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:30:04.525+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:30:04.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:30:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:30:04.743+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:30:04.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:30:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:30:04.743+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:30:04.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:30:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:30:04.743+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:30:04.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:30:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:30:04.743+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:30:05.507 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:30:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:30:05.208+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 6632 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:30:05.767 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:30:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:30:05.740+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:30:05.767 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:30:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:30:05.507+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:30:05.767 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:30:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:30:05.507+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:30:05.768 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:30:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:30:05.507+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:30:05.768 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:30:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:30:05.508+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:30:06.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:30:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:30:05.767+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:30:06.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:30:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:30:05.767+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:30:06.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:30:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:30:05.767+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:30:06.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:30:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:30:05.767+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:30:06.754 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:30:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:30:06.549+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:30:06.755 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:30:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:30:06.549+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:30:06.755 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:30:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:30:06.549+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:30:06.755 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:30:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:30:06.550+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:30:07.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:30:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:30:06.765+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:30:07.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:30:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:30:06.754+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:30:07.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:30:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:30:06.754+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:30:07.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:30:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:30:06.754+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:30:07.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:30:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:30:06.754+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:30:07.795 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:30:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:30:07.720+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:30:07.796 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:30:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:30:07.507+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:30:07.796 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:30:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:30:07.507+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:30:07.796 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:30:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:30:07.507+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:30:07.796 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:30:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:30:07.508+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:30:08.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:30:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:30:07.795+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:30:08.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:30:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:30:07.795+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:30:08.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:30:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:30:07.795+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:30:08.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:30:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:30:07.795+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:30:08.734 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:30:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:30:08.467+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:30:08.735 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:30:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:30:08.467+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:30:08.735 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:30:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:30:08.467+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:30:08.735 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:30:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:30:08.468+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:30:09.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:30:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:30:08.734+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:30:09.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:30:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:30:08.794+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:30:09.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:30:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:30:08.794+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:30:09.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:30:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:30:08.794+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:30:09.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:30:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:30:08.795+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:30:09.772 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:30:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:30:09.506+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:30:09.773 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:30:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:30:09.506+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:30:09.773 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:30:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:30:09.506+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:30:09.773 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:30:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:30:09.507+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:30:10.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:30:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:30:09.772+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:30:10.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:30:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:30:09.806+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:30:10.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:30:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:30:09.806+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:30:10.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:30:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:30:09.806+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:30:10.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:30:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:30:09.806+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:30:10.550 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:30:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:30:10.208+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 6725 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:30:10.805 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:30:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:30:10.763+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:30:10.806 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:30:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:30:10.549+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:30:10.806 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:30:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:30:10.549+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:30:10.806 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:30:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:30:10.549+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:30:10.806 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:30:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:30:10.550+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:30:11.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:30:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:30:10.805+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:30:11.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:30:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:30:10.805+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:30:11.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:30:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:30:10.805+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:30:11.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:30:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:30:10.805+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:30:11.355 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-18T01:30:11.353+0000 7fa8aec1f700 0 monclient(hunting): authenticate timed out after 300 2024-03-18T01:30:11.355 INFO:teuthology.orchestra.run.smithi029.stderr:[errno 110] RADOS timed out (error connecting to the cluster) 2024-03-18T01:30:11.662 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:30:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:30:11.563+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:30:11.662 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:30:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:30:11.563+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:30:11.662 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:30:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:30:11.563+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:30:11.662 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:30:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:30:11.563+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:30:12.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:30:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:30:11.768+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:30:12.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:30:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:30:11.808+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:30:12.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:30:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:30:11.808+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:30:12.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:30:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:30:11.808+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:30:12.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:30:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:30:11.809+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:30:12.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-18T01:30:12.167 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/parallel.py", line 56, in task p.spawn(_run_spawned, ctx, confg, taskname) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/parallel.py", line 64, in _run_spawned mgr = run_tasks.run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/sequential.py", line 47, in task mgr = run_tasks.run_one_task(taskname, ctx=ctx, config=confg) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/github.com_batrick_ceph_c382027d5c66ef4e0aa1131ea3e8080c658d3184/qa/tasks/cephadm.py", line 1423, in shell _shell(ctx, cluster_name, remote, File "/home/teuthworker/src/github.com_batrick_ceph_c382027d5c66ef4e0aa1131ea3e8080c658d3184/qa/tasks/cephadm.py", line 97, in _shell return remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi029 with status 1: "sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 37f51182-e4c4-11ee-95c9-87774f69a715 -e sha1=3d78f5ba07bf18880172c07d4d7b4ac00c9d30af -- bash -c 'ceph orch ps'" 2024-03-18T01:30:12.380 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=d23aae5f1e6446eba31826c0f3a4cce6 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/parallel.py", line 56, in task p.spawn(_run_spawned, ctx, confg, taskname) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/parallel.py", line 64, in _run_spawned mgr = run_tasks.run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/sequential.py", line 47, in task mgr = run_tasks.run_one_task(taskname, ctx=ctx, config=confg) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/github.com_batrick_ceph_c382027d5c66ef4e0aa1131ea3e8080c658d3184/qa/tasks/cephadm.py", line 1423, in shell _shell(ctx, cluster_name, remote, File "/home/teuthworker/src/github.com_batrick_ceph_c382027d5c66ef4e0aa1131ea3e8080c658d3184/qa/tasks/cephadm.py", line 97, in _shell return remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi029 with status 1: "sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 37f51182-e4c4-11ee-95c9-87774f69a715 -e sha1=3d78f5ba07bf18880172c07d4d7b4ac00c9d30af -- bash -c 'ceph orch ps'" 2024-03-18T01:30:12.383 DEBUG:teuthology.run_tasks:Unwinding manager kclient 2024-03-18T01:30:12.432 INFO:tasks.kclient:Unmounting kernel clients... 2024-03-18T01:30:12.432 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-03-18T01:30:12.433 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-18T01:30:12.454 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-03-18T01:30:12.454 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-18T01:30:12.509 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T01:30:12.817 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:30:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:30:12.741+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:30:12.818 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:30:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:30:12.535+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:30:12.818 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:30:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:30:12.535+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:30:12.818 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:30:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:30:12.535+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:30:12.818 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:30:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:30:12.537+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:30:13.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:30:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:30:12.816+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:30:13.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:30:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:30:12.816+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:30:13.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:30:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:30:12.816+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:30:13.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:30:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:30:12.817+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:30:13.802 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:30:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:30:13.761+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:30:13.803 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:30:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:30:13.571+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:39.004075+0000 front 2024-03-18T01:19:39.004120+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:30:13.803 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:30:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:30:13.571+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:39.004093+0000 front 2024-03-18T01:19:39.004194+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:30:13.803 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:30:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:30:13.571+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:39.004140+0000 front 2024-03-18T01:19:39.004249+0000 (oldest deadline 2024-03-18T01:20:04.304137+0000) 2024-03-18T01:30:13.803 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:30:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:30:13.571+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:30:14.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:30:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:30:13.802+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:30:14.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:30:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:30:13.802+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:30:14.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:30:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:30:13.802+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:30:14.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:30:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:30:13.802+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:30:14.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:30:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:30:14.529+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:30:14.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:30:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:30:14.733+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:30:14.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:30:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:30:14.760+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:30:14.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:30:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:30:14.760+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:30:14.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:30:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:30:14.760+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:30:14.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:30:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:30:14.761+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:30:15.336 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:30:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:30:15.209+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 6819 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:30:15.695 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:30:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:30:15.512+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:30:16.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:30:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:30:15.694+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:30:16.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:30:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:30:15.763+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-03-18T01:19:42.994753+0000 front 2024-03-18T01:19:42.994603+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:30:16.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:30:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:30:15.763+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-03-18T01:19:42.994648+0000 front 2024-03-18T01:19:42.994732+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:30:16.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:30:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:30:15.763+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-03-18T01:19:42.994706+0000 front 2024-03-18T01:19:42.994695+0000 (oldest deadline 2024-03-18T01:20:06.494633+0000) 2024-03-18T01:30:16.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:30:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:30:15.763+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:30:16.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:30:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:30:16.506+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:30:16.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:30:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:30:16.695+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:30:16.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:30:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:30:16.717+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:30:17.724 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:30:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:30:17.460+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:30:18.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:30:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:30:17.730+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:30:18.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:30:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:30:17.724+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:30:18.779 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:30:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:30:18.510+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:30:18.780 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:30:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:30:18.720+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:30:18.780 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:30:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:30:18.738+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:30:19.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:30:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:30:19.550+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:30:19.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:30:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:30:19.762+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:30:19.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:30:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:30:19.767+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:30:20.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:30:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:30:20.210+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 6912 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:30:21.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:30:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:30:20.809+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:30:21.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:30:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:30:20.585+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:30:21.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:30:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:30:20.727+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:30:22.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:30:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:30:21.620+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:30:22.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:30:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:30:21.704+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:30:22.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:30:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:30:21.783+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:30:23.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:30:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:30:22.580+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:30:23.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:30:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:30:22.713+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:30:23.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:30:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:30:22.744+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:30:23.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:30:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:30:23.768+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:30:23.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:30:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:30:23.593+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:30:23.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:30:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:30:23.754+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:30:24.923 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:30:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:30:24.750+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:30:24.923 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:30:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:30:24.743+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:30:24.924 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:30:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:30:24.576+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:30:25.556 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:30:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:30:25.212+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 7008 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:30:25.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:30:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:30:25.735+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:30:25.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:30:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:30:25.555+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:30:25.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:30:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:30:25.761+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:30:25.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:30:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:30:25.761+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:30:25.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:30:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:30:25.761+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:30:25.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:30:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:30:25.762+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:30:26.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:30:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:30:26.708+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:30:26.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:30:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:30:26.538+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:30:26.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:30:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:30:26.714+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:30:26.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:30:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:30:26.714+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:30:26.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:30:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:30:26.714+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:30:26.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:30:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:30:26.715+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:30:27.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:30:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:30:27.669+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:30:27.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:30:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:30:27.498+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:30:27.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:30:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:30:27.702+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:30:27.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:30:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:30:27.702+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:30:27.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:30:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:30:27.702+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:30:27.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:30:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:30:27.703+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:30:28.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:30:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:30:28.494+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:30:28.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:30:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:30:28.703+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:30:28.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:30:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:30:28.675+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:30:28.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:30:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:30:28.675+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:30:28.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:30:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:30:28.675+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:30:28.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:30:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:30:28.676+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:30:29.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:30:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:30:29.654+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:30:29.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:30:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:30:29.492+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:30:29.821 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:30:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:30:29.662+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:30:29.821 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:30:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:30:29.662+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:30:29.821 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:30:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:30:29.662+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:30:29.822 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:30:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:30:29.663+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:30:30.543 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:30:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:30:30.213+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 7102 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:30:30.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:30:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:30:30.543+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:30:30.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:30:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:30:30.698+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:30:30.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:30:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:30:30.623+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:30:30.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:30:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:30:30.623+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:30:30.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:30:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:30:30.623+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:30:30.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:30:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:30:30.623+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:30:32.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:30:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:30:31.576+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:30:32.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:30:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:30:31.682+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:30:32.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:30:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:30:31.642+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:30:32.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:30:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:30:31.642+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:30:32.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:30:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:30:31.642+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:30:32.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:30:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:30:31.643+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:30:33.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:30:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:30:32.585+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:30:33.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:30:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:30:32.636+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:30:33.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:30:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:30:32.608+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:30:33.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:30:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:30:32.608+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:30:33.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:30:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:30:32.608+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:30:33.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:30:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:30:32.609+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:30:33.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:30:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:30:33.624+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:30:33.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:30:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:30:33.680+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:30:33.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:30:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:30:33.589+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:30:33.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:30:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:30:33.589+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:30:33.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:30:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:30:33.589+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:30:33.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:30:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:30:33.590+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:30:34.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:30:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:30:34.670+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:30:34.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:30:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:30:34.720+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:30:34.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:30:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:30:34.568+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:30:34.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:30:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:30:34.568+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:30:34.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:30:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:30:34.568+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:30:34.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:30:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:30:34.569+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:30:35.531 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:30:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:30:35.214+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 7198 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:30:35.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:30:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:30:35.680+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:30:35.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:30:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:30:35.530+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:30:35.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:30:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:30:35.530+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:30:35.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:30:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:30:35.530+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:30:35.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:30:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:30:35.531+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:30:35.821 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:30:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:30:35.677+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:30:35.821 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:30:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:30:35.677+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:30:35.821 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:30:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:30:35.677+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:30:35.821 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:30:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:30:35.677+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:30:37.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:30:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:30:36.699+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:30:37.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:30:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:30:36.570+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:30:37.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:30:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:30:36.570+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:30:37.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:30:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:30:36.570+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:30:37.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:30:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:30:36.570+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:30:37.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:30:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:30:36.685+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:30:37.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:30:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:30:36.685+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:30:37.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:30:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:30:36.685+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:30:37.071 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:30:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:30:36.685+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:30:37.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:30:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:30:37.663+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:30:37.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:30:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:30:37.702+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:30:37.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:30:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:30:37.702+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:30:37.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:30:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:30:37.702+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:30:37.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:30:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:30:37.703+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:30:37.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:30:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:30:37.563+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:30:37.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:30:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:30:37.563+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:30:37.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:30:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:30:37.563+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:30:37.821 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:30:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:30:37.564+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:30:38.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:30:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:30:38.694+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:30:38.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:30:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:30:38.698+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:30:38.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:30:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:30:38.698+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:30:38.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:30:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:30:38.698+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:30:38.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:30:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:30:38.698+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:30:38.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:30:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:30:38.521+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:30:38.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:30:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:30:38.521+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:30:38.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:30:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:30:38.521+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:30:38.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:30:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:30:38.521+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:30:39.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:30:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:30:39.689+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:30:39.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:30:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:30:39.651+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:30:39.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:30:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:30:39.651+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:30:39.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:30:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:30:39.651+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:30:39.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:30:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:30:39.652+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:30:39.821 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:30:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:30:39.520+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:30:39.821 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:30:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:30:39.520+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:30:39.821 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:30:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:30:39.520+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:30:39.821 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:30:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:30:39.520+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:30:40.563 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:30:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:30:40.214+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 7291 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:30:40.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:30:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:30:40.647+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:30:40.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:30:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:30:40.647+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:30:40.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:30:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:30:40.647+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:30:40.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:30:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:30:40.647+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:30:40.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:30:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:30:40.637+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:30:40.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:30:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:30:40.637+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:30:40.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:30:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:30:40.637+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:30:40.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:30:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:30:40.638+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:30:40.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:30:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:30:40.563+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:30:40.821 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:30:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:30:40.563+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:30:40.821 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:30:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:30:40.563+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:30:40.821 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:30:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:30:40.563+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:30:42.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:30:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:30:41.696+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:30:42.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:30:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:30:41.696+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:30:42.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:30:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:30:41.696+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:30:42.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:30:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:30:41.697+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:30:42.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:30:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:30:41.591+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:30:42.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:30:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:30:41.591+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:30:42.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:30:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:30:41.591+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:30:42.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:30:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:30:41.592+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:30:42.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:30:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:30:41.583+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:30:42.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:30:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:30:41.583+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:30:42.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:30:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:30:41.583+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:30:42.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:30:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:30:41.584+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:30:43.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:30:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:30:42.726+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:30:43.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:30:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:30:42.726+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:30:43.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:30:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:30:42.726+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:30:43.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:30:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:30:42.727+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:30:43.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:30:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:30:42.601+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:30:43.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:30:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:30:42.601+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:30:43.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:30:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:30:42.601+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:30:43.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:30:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:30:42.602+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:30:43.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:30:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:30:42.623+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:30:43.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:30:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:30:42.623+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:30:43.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:30:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:30:42.623+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:30:43.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:30:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:30:42.624+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:30:44.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:30:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:30:43.607+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:30:44.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:30:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:30:43.607+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:30:44.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:30:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:30:43.607+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:30:44.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:30:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:30:43.608+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:30:44.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:30:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:30:43.631+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:30:44.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:30:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:30:43.631+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:30:44.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:30:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:30:43.631+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:30:44.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:30:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:30:43.632+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:30:44.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:30:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:30:43.726+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:30:44.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:30:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:30:43.726+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:30:44.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:30:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:30:43.726+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:30:44.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:30:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:30:43.726+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:30:45.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:30:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:30:44.728+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:30:45.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:30:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:30:44.728+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:30:45.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:30:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:30:44.728+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:30:45.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:30:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:30:44.728+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:30:45.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:30:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:30:44.644+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:30:45.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:30:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:30:44.644+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:30:45.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:30:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:30:44.644+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:30:45.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:30:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:30:44.645+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:30:45.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:30:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:30:44.635+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:30:45.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:30:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:30:44.635+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:30:45.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:30:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:30:44.635+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:30:45.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:30:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:30:44.636+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:30:45.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:30:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:30:45.215+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 7387 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:30:46.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:30:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:30:45.632+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:30:46.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:30:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:30:45.632+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:30:46.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:30:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:30:45.633+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:30:46.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:30:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:30:45.634+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:30:46.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:30:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:30:45.608+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:30:46.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:30:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:30:45.608+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:30:46.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:30:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:30:45.608+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:30:46.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:30:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:30:45.609+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:30:46.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:30:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:30:45.764+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:30:46.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:30:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:30:45.764+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:30:46.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:30:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:30:45.764+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:30:46.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:30:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:30:45.764+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:30:46.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:30:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:30:46.648+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:30:46.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:30:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:30:46.649+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:30:46.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:30:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:30:46.649+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:30:46.724 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:30:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:30:46.649+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:30:46.724 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:30:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:30:46.650+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:30:46.724 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:30:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:30:46.650+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:30:46.724 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:30:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:30:46.650+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:30:46.724 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:30:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:30:46.651+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:30:47.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:30:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:30:46.722+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:30:47.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:30:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:30:46.722+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:30:47.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:30:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:30:46.722+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:30:47.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:30:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:30:46.723+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:30:48.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:30:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:30:47.765+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:30:48.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:30:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:30:47.765+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:30:48.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:30:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:30:47.765+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:30:48.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:30:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:30:47.766+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:30:48.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:30:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:30:47.657+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:30:48.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:30:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:30:47.657+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:30:48.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:30:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:30:47.657+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:30:48.071 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:30:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:30:47.658+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:30:48.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:30:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:30:47.655+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:30:48.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:30:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:30:47.655+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:30:48.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:30:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:30:47.655+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:30:48.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:30:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:30:47.655+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:30:49.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:30:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:30:48.647+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:30:49.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:30:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:30:48.647+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:30:49.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:30:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:30:48.647+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:30:49.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:30:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:30:48.648+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:30:49.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:30:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:30:48.683+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:30:49.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:30:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:30:48.683+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:30:49.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:30:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:30:48.683+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:30:49.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:30:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:30:48.684+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:30:49.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:30:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:30:48.728+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:30:49.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:30:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:30:48.728+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:30:49.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:30:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:30:48.728+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:30:49.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:30:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:30:48.729+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:30:50.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:30:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:30:49.745+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:30:50.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:30:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:30:49.745+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:30:50.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:30:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:30:49.745+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:30:50.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:30:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:30:49.745+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:30:50.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:30:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:30:49.733+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:30:50.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:30:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:30:49.733+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:30:50.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:30:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:30:49.733+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:30:50.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:30:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:30:49.734+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:30:50.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:30:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:30:49.615+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:30:50.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:30:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:30:49.615+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:30:50.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:30:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:30:49.615+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:30:50.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:30:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:30:49.616+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:30:50.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:30:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:30:50.216+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 7481 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:30:51.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:30:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:30:50.700+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:30:51.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:30:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:30:50.700+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:30:51.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:30:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:30:50.700+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:30:51.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:30:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:30:50.700+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:30:51.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:30:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:30:50.760+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:30:51.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:30:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:30:50.760+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:30:51.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:30:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:30:50.760+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:30:51.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:30:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:30:50.761+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:30:51.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:30:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:30:50.617+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:30:51.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:30:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:30:50.617+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:30:51.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:30:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:30:50.617+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:30:51.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:30:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:30:50.618+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:30:52.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:30:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:30:51.580+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:30:52.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:30:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:30:51.580+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:30:52.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:30:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:30:51.580+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:30:52.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:30:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:30:51.581+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:30:52.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:30:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:30:51.736+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:30:52.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:30:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:30:51.736+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:30:52.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:30:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:30:51.736+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:30:52.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:30:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:30:51.737+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:30:52.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:30:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:30:51.703+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:30:52.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:30:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:30:51.703+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:30:52.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:30:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:30:51.703+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:30:52.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:30:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:30:51.704+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:30:52.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:30:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:30:52.560+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:30:52.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:30:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:30:52.560+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:30:52.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:30:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:30:52.560+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:30:52.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:30:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:30:52.561+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:30:52.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:30:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:30:52.688+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:30:52.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:30:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:30:52.688+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:30:52.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:30:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:30:52.688+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:30:52.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:30:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:30:52.688+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:30:52.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:30:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:30:52.729+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:30:52.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:30:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:30:52.729+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:30:52.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:30:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:30:52.729+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:30:52.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:30:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:30:52.729+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:30:53.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:30:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:30:53.585+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:30:53.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:30:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:30:53.585+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:30:53.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:30:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:30:53.585+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:30:53.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:30:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:30:53.586+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:30:53.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:30:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:30:53.707+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:30:53.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:30:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:30:53.707+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:30:53.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:30:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:30:53.707+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:30:53.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:30:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:30:53.707+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:30:53.821 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:30:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:30:53.665+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:30:53.821 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:30:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:30:53.665+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:30:53.821 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:30:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:30:53.665+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:30:53.821 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:30:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:30:53.666+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:30:54.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:30:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:30:54.707+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:30:54.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:30:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:30:54.707+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:30:54.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:30:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:30:54.707+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:30:54.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:30:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:30:54.708+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:30:54.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:30:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:30:54.710+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:30:54.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:30:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:30:54.710+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:30:54.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:30:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:30:54.710+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:30:54.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:30:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:30:54.710+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:30:54.821 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:30:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:30:54.554+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:30:54.821 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:30:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:30:54.554+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:30:54.821 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:30:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:30:54.554+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:30:54.821 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:30:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:30:54.554+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:30:55.535 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:30:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:30:55.216+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 7576 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:30:55.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:30:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:30:55.721+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:30:55.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:30:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:30:55.721+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:30:55.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:30:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:30:55.721+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:30:55.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:30:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:30:55.722+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:30:55.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:30:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:30:55.534+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:30:55.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:30:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:30:55.534+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:30:55.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:30:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:30:55.534+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:30:55.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:30:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:30:55.535+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:30:55.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:30:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:30:55.731+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:30:55.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:30:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:30:55.731+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:30:55.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:30:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:30:55.731+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:30:55.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:30:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:30:55.732+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:30:56.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:30:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:30:56.527+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:30:56.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:30:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:30:56.527+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:30:56.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:30:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:30:56.527+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:30:56.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:30:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:30:56.528+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:30:56.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:30:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:30:56.697+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:30:56.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:30:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:30:56.697+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:30:56.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:30:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:30:56.697+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:30:56.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:30:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:30:56.698+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:30:56.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:30:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:30:56.776+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:30:56.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:30:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:30:56.776+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:30:56.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:30:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:30:56.776+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:30:56.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:30:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:30:56.776+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:30:57.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:30:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:30:57.551+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:30:57.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:30:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:30:57.551+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:30:57.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:30:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:30:57.551+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:30:57.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:30:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:30:57.552+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:30:57.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:30:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:30:57.655+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:30:57.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:30:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:30:57.655+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:30:57.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:30:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:30:57.655+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:30:57.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:30:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:30:57.656+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:30:57.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:30:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:30:57.756+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:30:57.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:30:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:30:57.756+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:30:57.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:30:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:30:57.756+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:30:57.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:30:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:30:57.756+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:30:58.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:30:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:30:58.560+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:30:58.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:30:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:30:58.560+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:30:58.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:30:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:30:58.560+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:30:58.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:30:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:30:58.560+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:30:58.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:30:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:30:58.740+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:30:58.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:30:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:30:58.740+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:30:58.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:30:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:30:58.740+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:30:58.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:30:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:30:58.740+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:30:58.821 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:30:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:30:58.659+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:30:58.821 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:30:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:30:58.659+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:30:58.821 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:30:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:30:58.659+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:30:58.822 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:30:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:30:58.660+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:30:59.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:30:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:30:59.622+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:30:59.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:30:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:30:59.622+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:30:59.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:30:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:30:59.622+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:30:59.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:30:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:30:59.623+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:30:59.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:30:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:30:59.546+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:30:59.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:30:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:30:59.546+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:30:59.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:30:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:30:59.546+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:30:59.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:30:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:30:59.546+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:30:59.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:30:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:30:59.739+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:30:59.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:30:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:30:59.739+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:30:59.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:30:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:30:59.739+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:30:59.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:30:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:30:59.739+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:31:00.566 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:31:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:31:00.217+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 7669 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:31:00.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:31:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:31:00.635+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:31:00.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:31:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:31:00.635+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:31:00.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:31:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:31:00.635+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:31:00.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:31:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:31:00.635+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:31:00.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:31:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:31:00.564+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:31:00.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:31:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:31:00.564+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:31:00.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:31:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:31:00.564+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:31:00.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:31:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:31:00.565+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:31:00.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:31:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:31:00.731+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:31:00.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:31:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:31:00.731+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:31:00.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:31:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:31:00.731+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:31:00.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:31:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:31:00.731+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:31:01.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:31:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:31:01.639+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:31:01.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:31:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:31:01.639+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:31:01.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:31:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:31:01.639+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:31:01.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:31:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:31:01.640+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:31:01.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:31:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:31:01.555+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:31:01.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:31:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:31:01.555+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:31:01.821 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:31:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:31:01.555+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:31:01.821 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:31:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:31:01.556+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:31:01.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:31:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:31:01.692+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:31:01.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:31:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:31:01.692+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:31:01.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:31:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:31:01.692+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:31:01.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:31:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:31:01.692+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:31:03.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:31:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:31:02.735+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:31:03.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:31:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:31:02.735+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:31:03.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:31:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:31:02.735+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:31:03.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:31:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:31:02.735+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:31:03.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:31:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:31:02.572+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:31:03.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:31:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:31:02.572+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:31:03.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:31:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:31:02.572+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:31:03.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:31:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:31:02.573+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:31:03.071 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:31:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:31:02.606+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:31:03.071 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:31:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:31:02.606+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:31:03.071 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:31:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:31:02.606+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:31:03.072 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:31:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:31:02.607+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:31:03.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:31:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:31:03.655+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:31:03.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:31:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:31:03.655+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:31:03.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:31:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:31:03.655+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:31:03.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:31:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:31:03.656+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:31:03.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:31:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:31:03.624+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:31:03.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:31:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:31:03.624+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:31:03.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:31:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:31:03.624+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:31:03.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:31:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:31:03.625+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:31:03.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:31:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:31:03.724+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:31:03.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:31:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:31:03.724+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:31:03.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:31:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:31:03.724+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:31:03.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:31:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:31:03.724+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:31:04.859 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:31:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:31:04.710+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:31:04.860 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:31:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:31:04.710+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:31:04.860 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:31:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:31:04.710+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:31:04.860 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:31:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:31:04.711+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:31:04.860 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:31:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:31:04.615+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:31:04.860 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:31:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:31:04.615+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:31:04.861 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:31:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:31:04.615+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:31:04.861 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:31:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:31:04.616+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:31:04.861 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:31:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:31:04.576+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:31:04.861 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:31:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:31:04.576+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:31:04.861 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:31:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:31:04.576+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:31:04.861 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:31:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:31:04.576+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:31:05.562 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:31:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:31:05.219+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 7765 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:31:05.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:31:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:31:05.707+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:31:05.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:31:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:31:05.707+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:31:05.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:31:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:31:05.707+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:31:05.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:31:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:31:05.708+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:31:05.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:31:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:31:05.635+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:31:05.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:31:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:31:05.635+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:31:05.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:31:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:31:05.635+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:31:05.821 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:31:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:31:05.636+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:31:05.821 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:31:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:31:05.561+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:31:05.821 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:31:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:31:05.561+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:31:05.821 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:31:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:31:05.561+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:31:05.821 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:31:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:31:05.563+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:31:06.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:31:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:31:06.620+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:31:06.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:31:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:31:06.620+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:31:06.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:31:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:31:06.620+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:31:06.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:31:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:31:06.621+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:31:06.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:31:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:31:06.562+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:31:06.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:31:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:31:06.562+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:31:06.821 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:31:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:31:06.562+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:31:06.821 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:31:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:31:06.562+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:31:06.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:31:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:31:06.681+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:31:06.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:31:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:31:06.681+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:31:06.822 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:31:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:31:06.681+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:31:06.822 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:31:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:31:06.682+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:31:07.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:31:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:31:07.607+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:31:07.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:31:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:31:07.607+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:31:07.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:31:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:31:07.607+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:31:07.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:31:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:31:07.608+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:31:07.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:31:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:31:07.545+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:31:07.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:31:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:31:07.545+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:31:07.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:31:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:31:07.545+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:31:07.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:31:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:31:07.545+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:31:07.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:31:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:31:07.648+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:31:07.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:31:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:31:07.648+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:31:07.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:31:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:31:07.648+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:31:07.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:31:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:31:07.648+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:31:08.577 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:31:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:31:08.540+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:31:08.577 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:31:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:31:08.540+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:31:08.577 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:31:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:31:08.540+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:31:08.578 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:31:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:31:08.541+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:31:09.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:31:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:31:08.576+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:31:09.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:31:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:31:08.576+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:31:09.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:31:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:31:08.576+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:31:09.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:31:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:31:08.577+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:31:09.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:31:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:31:08.688+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:31:09.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:31:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:31:08.688+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:31:09.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:31:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:31:08.688+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:31:09.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:31:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:31:08.688+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:31:10.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:31:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:31:09.586+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:31:10.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:31:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:31:09.586+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:31:10.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:31:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:31:09.586+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:31:10.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:31:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:31:09.587+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:31:10.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:31:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:31:09.571+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:31:10.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:31:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:31:09.571+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:31:10.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:31:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:31:09.571+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:31:10.071 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:31:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:31:09.572+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:31:10.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:31:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:31:09.668+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:31:10.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:31:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:31:09.668+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:31:10.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:31:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:31:09.668+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:31:10.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:31:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:31:09.668+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:31:10.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:31:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:31:10.220+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 7859 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:31:11.004 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:31:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:31:10.585+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:31:11.004 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:31:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:31:10.585+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:31:11.004 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:31:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:31:10.585+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:31:11.004 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:31:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:31:10.586+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:31:11.005 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:31:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:31:10.583+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:31:11.005 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:31:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:31:10.583+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:31:11.005 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:31:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:31:10.583+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:31:11.005 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:31:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:31:10.583+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:31:11.005 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:31:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:31:10.641+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:31:11.006 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:31:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:31:10.641+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:31:11.006 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:31:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:31:10.641+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:31:11.006 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:31:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:31:10.642+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:31:12.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:31:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:31:11.597+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:31:12.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:31:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:31:11.597+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:31:12.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:31:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:31:11.597+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:31:12.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:31:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:31:11.598+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:31:12.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:31:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:31:11.626+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:31:12.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:31:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:31:11.626+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:31:12.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:31:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:31:11.626+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:31:12.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:31:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:31:11.627+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:31:12.071 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:31:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:31:11.573+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:31:12.071 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:31:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:31:11.573+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:31:12.071 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:31:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:31:11.573+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:31:12.071 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:31:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:31:11.574+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:31:12.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:31:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:31:12.615+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:31:12.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:31:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:31:12.615+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:31:12.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:31:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:31:12.615+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:31:12.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:31:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:31:12.615+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:31:12.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:31:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:31:12.547+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:31:12.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:31:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:31:12.547+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:31:12.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:31:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:31:12.547+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:31:12.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:31:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:31:12.548+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:31:12.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:31:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:31:12.583+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:31:12.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:31:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:31:12.583+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:31:12.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:31:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:31:12.583+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:31:12.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:31:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:31:12.583+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:31:13.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:31:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:31:13.587+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:31:13.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:31:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:31:13.587+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:31:13.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:31:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:31:13.587+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:31:13.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:31:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:31:13.588+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:31:13.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:31:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:31:13.631+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:31:13.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:31:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:31:13.631+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:31:13.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:31:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:31:13.631+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:31:13.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:31:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:31:13.632+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:31:13.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:31:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:31:13.546+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:31:13.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:31:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:31:13.546+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:31:13.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:31:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:31:13.546+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:31:13.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:31:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:31:13.546+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:31:14.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:31:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:31:14.530+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:31:14.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:31:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:31:14.530+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:31:14.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:31:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:31:14.530+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:31:14.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:31:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:31:14.530+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:31:14.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:31:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:31:14.619+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:31:14.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:31:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:31:14.619+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:31:14.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:31:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:31:14.619+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:31:14.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:31:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:31:14.620+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:31:14.821 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:31:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:31:14.624+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:31:14.821 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:31:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:31:14.624+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:31:14.821 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:31:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:31:14.624+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:31:14.821 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:31:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:31:14.625+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:31:15.501 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:31:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:31:15.222+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 7954 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:31:15.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:31:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:31:15.654+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:31:15.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:31:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:31:15.654+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:31:15.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:31:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:31:15.654+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:31:15.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:31:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:31:15.655+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:31:15.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:31:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:31:15.500+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:31:15.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:31:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:31:15.500+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:31:15.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:31:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:31:15.500+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:31:15.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:31:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:31:15.500+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:31:15.821 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:31:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:31:15.587+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:31:15.821 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:31:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:31:15.587+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:31:15.821 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:31:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:31:15.587+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:31:15.821 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:31:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:31:15.588+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:31:16.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:31:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:31:16.686+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:31:16.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:31:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:31:16.686+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:31:16.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:31:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:31:16.686+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:31:16.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:31:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:31:16.687+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:31:16.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:31:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:31:16.490+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:31:16.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:31:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:31:16.490+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:31:16.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:31:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:31:16.490+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:31:16.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:31:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:31:16.490+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:31:16.821 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:31:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:31:16.593+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:31:16.822 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:31:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:31:16.593+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:31:16.822 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:31:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:31:16.593+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:31:16.822 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:31:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:31:16.594+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:31:17.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:31:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:31:17.498+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:31:17.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:31:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:31:17.498+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:31:17.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:31:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:31:17.498+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:31:17.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:31:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:31:17.498+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:31:17.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:31:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:31:17.567+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:31:17.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:31:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:31:17.567+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:31:17.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:31:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:31:17.567+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:31:17.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:31:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:31:17.568+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:31:17.821 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:31:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:31:17.724+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:31:17.821 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:31:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:31:17.724+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:31:17.821 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:31:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:31:17.724+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:31:17.821 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:31:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:31:17.725+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:31:18.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:31:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:31:18.526+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:31:18.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:31:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:31:18.526+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:31:18.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:31:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:31:18.526+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:31:18.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:31:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:31:18.527+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:31:18.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:31:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:31:18.681+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:31:18.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:31:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:31:18.681+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:31:18.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:31:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:31:18.681+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:31:18.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:31:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:31:18.682+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:31:18.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:31:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:31:18.517+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:31:18.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:31:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:31:18.517+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:31:18.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:31:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:31:18.517+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:31:18.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:31:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:31:18.517+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:31:19.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:31:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:31:19.635+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:31:19.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:31:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:31:19.635+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:31:19.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:31:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:31:19.635+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:31:19.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:31:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:31:19.636+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:31:19.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:31:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:31:19.544+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:31:19.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:31:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:31:19.544+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:31:19.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:31:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:31:19.544+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:31:19.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:31:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:31:19.544+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:31:19.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:31:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:31:19.574+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:31:19.821 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:31:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:31:19.574+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:31:19.821 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:31:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:31:19.574+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:31:19.821 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:31:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:31:19.575+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:31:20.502 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:31:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:31:20.223+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 8047 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:31:20.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:31:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:31:20.631+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:31:20.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:31:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:31:20.631+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:31:20.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:31:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:31:20.631+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:31:20.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:31:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:31:20.631+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:31:20.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:31:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:31:20.530+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:31:20.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:31:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:31:20.530+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:31:20.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:31:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:31:20.530+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:31:20.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:31:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:31:20.531+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:31:20.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:31:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:31:20.501+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:31:20.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:31:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:31:20.501+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:31:20.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:31:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:31:20.501+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:31:20.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:31:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:31:20.501+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:31:21.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:31:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:31:21.511+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:31:21.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:31:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:31:21.511+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:31:21.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:31:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:31:21.511+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:31:21.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:31:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:31:21.512+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:31:21.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:31:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:31:21.600+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:31:21.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:31:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:31:21.600+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:31:21.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:31:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:31:21.600+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:31:21.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:31:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:31:21.601+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:31:21.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:31:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:31:21.542+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:31:21.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:31:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:31:21.542+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:31:21.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:31:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:31:21.542+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:31:21.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:31:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:31:21.542+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:31:22.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:31:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:31:22.536+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:31:22.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:31:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:31:22.536+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:31:22.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:31:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:31:22.536+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:31:22.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:31:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:31:22.537+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:31:22.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:31:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:31:22.565+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:31:22.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:31:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:31:22.565+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:31:22.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:31:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:31:22.565+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:31:22.821 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:31:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:31:22.566+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:31:22.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:31:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:31:22.591+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:31:22.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:31:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:31:22.591+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:31:22.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:31:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:31:22.591+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:31:22.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:31:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:31:22.591+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:31:23.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:31:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:31:23.611+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:31:23.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:31:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:31:23.611+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:31:23.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:31:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:31:23.611+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:31:23.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:31:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:31:23.612+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:31:23.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:31:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:31:23.579+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:31:23.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:31:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:31:23.579+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:31:23.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:31:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:31:23.579+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:31:23.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:31:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:31:23.580+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:31:23.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:31:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:31:23.602+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:31:23.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:31:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:31:23.602+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:31:23.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:31:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:31:23.602+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:31:23.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:31:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:31:23.603+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:31:24.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:31:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:31:24.643+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:31:24.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:31:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:31:24.643+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:31:24.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:31:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:31:24.643+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:31:24.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:31:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:31:24.643+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:31:24.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:31:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:31:24.573+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:31:24.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:31:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:31:24.573+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:31:24.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:31:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:31:24.573+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:31:24.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:31:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:31:24.574+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:31:24.821 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:31:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:31:24.548+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:31:24.821 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:31:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:31:24.548+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:31:24.821 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:31:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:31:24.548+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:31:24.821 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:31:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:31:24.549+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:31:25.502 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:31:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:31:25.224+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 8142 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:31:25.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:31:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:31:25.501+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:31:25.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:31:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:31:25.501+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:31:25.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:31:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:31:25.501+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:31:25.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:31:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:31:25.502+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:31:25.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:31:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:31:25.588+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:31:25.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:31:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:31:25.588+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:31:25.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:31:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:31:25.588+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:31:25.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:31:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:31:25.588+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:31:25.821 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:31:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:31:25.668+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:31:25.821 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:31:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:31:25.668+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:31:25.821 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:31:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:31:25.668+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:31:25.821 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:31:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:31:25.669+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:31:26.619 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:31:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:31:26.548+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:31:26.620 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:31:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:31:26.548+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:31:26.620 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:31:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:31:26.548+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:31:26.620 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:31:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:31:26.549+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:31:27.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:31:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:31:26.644+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:31:27.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:31:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:31:26.644+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:31:27.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:31:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:31:26.644+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:31:27.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:31:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:31:26.645+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:31:27.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:31:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:31:26.618+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:31:27.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:31:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:31:26.618+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:31:27.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:31:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:31:26.619+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:31:27.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:31:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:31:26.619+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:31:27.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:31:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:31:27.512+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:31:27.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:31:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:31:27.512+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:31:27.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:31:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:31:27.512+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:31:27.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:31:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:31:27.514+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:31:27.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:31:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:31:27.598+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:31:27.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:31:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:31:27.598+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:31:27.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:31:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:31:27.598+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:31:27.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:31:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:31:27.599+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:31:27.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:31:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:31:27.583+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:31:27.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:31:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:31:27.583+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:31:27.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:31:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:31:27.583+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:31:27.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:31:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:31:27.583+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:31:28.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:31:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:31:28.557+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:31:28.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:31:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:31:28.557+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:31:28.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:31:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:31:28.557+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:31:28.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:31:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:31:28.558+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:31:28.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:31:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:31:28.619+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:31:28.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:31:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:31:28.619+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:31:28.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:31:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:31:28.619+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:31:28.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:31:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:31:28.620+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:31:28.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:31:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:31:28.589+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:31:28.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:31:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:31:28.589+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:31:28.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:31:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:31:28.589+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:31:28.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:31:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:31:28.590+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:31:30.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:31:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:31:29.569+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:31:30.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:31:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:31:29.569+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:31:30.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:31:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:31:29.569+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:31:30.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:31:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:31:29.570+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:31:30.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:31:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:31:29.599+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:31:30.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:31:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:31:29.599+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:31:30.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:31:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:31:29.599+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:31:30.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:31:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:31:29.600+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:31:30.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:31:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:31:29.634+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:31:30.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:31:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:31:29.634+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:31:30.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:31:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:31:29.634+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:31:30.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:31:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:31:29.635+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:31:30.547 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:31:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:31:30.225+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 8235 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:31:30.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:31:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:31:30.636+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:31:30.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:31:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:31:30.636+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:31:30.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:31:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:31:30.636+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:31:30.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:31:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:31:30.637+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:31:30.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:31:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:31:30.547+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:31:30.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:31:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:31:30.547+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:31:30.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:31:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:31:30.547+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:31:30.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:31:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:31:30.547+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:31:30.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:31:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:31:30.674+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:31:30.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:31:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:31:30.674+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:31:30.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:31:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:31:30.674+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:31:30.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:31:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:31:30.675+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:31:32.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:31:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:31:31.574+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:31:32.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:31:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:31:31.574+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:31:32.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:31:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:31:31.574+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:31:32.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:31:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:31:31.574+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:31:32.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:31:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:31:31.620+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:31:32.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:31:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:31:31.620+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:31:32.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:31:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:31:31.620+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:31:32.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:31:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:31:31.620+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:31:32.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:31:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:31:31.626+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:31:32.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:31:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:31:31.626+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:31:32.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:31:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:31:31.626+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:31:32.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:31:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:31:31.627+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:31:32.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:31:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:31:32.562+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:31:32.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:31:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:31:32.562+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:31:32.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:31:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:31:32.562+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:31:32.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:31:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:31:32.563+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:31:32.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:31:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:31:32.658+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:31:32.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:31:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:31:32.658+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:31:32.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:31:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:31:32.658+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:31:32.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:31:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:31:32.658+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:31:32.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:31:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:31:32.581+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:31:32.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:31:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:31:32.581+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:31:32.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:31:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:31:32.581+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:31:32.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:31:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:31:32.581+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:31:33.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:31:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:31:33.613+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:31:33.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:31:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:31:33.613+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:31:33.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:31:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:31:33.613+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:31:33.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:31:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:31:33.613+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:31:33.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:31:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:31:33.628+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:31:33.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:31:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:31:33.628+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:31:33.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:31:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:31:33.628+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:31:33.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:31:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:31:33.628+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:31:33.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:31:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:31:33.631+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:31:33.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:31:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:31:33.631+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:31:33.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:31:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:31:33.631+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:31:33.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:31:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:31:33.631+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:31:34.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:31:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:31:34.567+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:31:34.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:31:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:31:34.567+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:31:34.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:31:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:31:34.567+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:31:34.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:31:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:31:34.568+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:31:34.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:31:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:31:34.603+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:31:34.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:31:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:31:34.603+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:31:34.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:31:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:31:34.603+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:31:34.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:31:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:31:34.604+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:31:34.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:31:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:31:34.619+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:31:34.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:31:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:31:34.619+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:31:34.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:31:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:31:34.619+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:31:34.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:31:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:31:34.620+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:31:35.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:31:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:31:35.226+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 8329 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:31:36.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:31:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:31:35.592+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:31:36.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:31:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:31:35.592+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:31:36.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:31:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:31:35.592+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:31:36.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:31:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:31:35.592+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:31:36.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:31:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:31:35.612+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:31:36.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:31:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:31:35.612+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:31:36.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:31:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:31:35.612+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:31:36.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:31:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:31:35.613+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:31:36.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:31:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:31:35.615+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:31:36.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:31:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:31:35.615+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:31:36.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:31:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:31:35.616+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:31:36.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:31:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:31:35.616+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:31:36.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:31:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:31:36.632+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:31:36.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:31:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:31:36.632+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:31:36.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:31:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:31:36.632+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:31:36.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:31:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:31:36.633+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:31:36.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:31:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:31:36.554+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:31:36.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:31:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:31:36.554+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:31:36.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:31:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:31:36.554+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:31:36.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:31:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:31:36.554+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:31:36.821 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:31:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:31:36.598+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:31:36.821 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:31:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:31:36.598+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:31:36.821 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:31:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:31:36.598+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:31:36.821 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:31:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:31:36.599+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:31:37.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:31:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:31:37.634+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:31:37.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:31:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:31:37.634+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:31:37.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:31:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:31:37.634+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:31:37.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:31:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:31:37.634+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:31:37.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:31:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:31:37.645+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:31:37.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:31:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:31:37.645+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:31:37.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:31:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:31:37.645+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:31:37.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:31:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:31:37.647+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:31:37.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:31:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:31:37.549+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:31:37.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:31:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:31:37.549+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:31:37.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:31:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:31:37.549+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:31:37.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:31:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:31:37.550+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:31:38.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:31:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:31:38.654+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:31:38.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:31:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:31:38.654+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:31:38.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:31:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:31:38.654+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:31:38.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:31:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:31:38.655+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:31:38.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:31:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:31:38.641+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:31:38.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:31:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:31:38.641+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:31:38.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:31:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:31:38.641+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:31:38.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:31:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:31:38.641+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:31:38.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:31:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:31:38.543+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:31:38.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:31:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:31:38.543+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:31:38.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:31:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:31:38.543+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:31:38.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:31:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:31:38.543+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:31:40.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:31:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:31:39.568+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:31:40.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:31:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:31:39.568+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:31:40.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:31:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:31:39.568+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:31:40.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:31:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:31:39.569+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:31:40.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:31:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:31:39.632+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:31:40.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:31:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:31:39.632+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:31:40.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:31:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:31:39.632+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:31:40.071 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:31:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:31:39.632+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:31:40.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:31:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:31:39.652+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:31:40.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:31:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:31:39.652+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:31:40.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:31:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:31:39.653+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:31:40.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:31:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:31:39.653+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:31:40.522 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:31:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:31:40.227+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 8421 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:31:40.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:31:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:31:40.626+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:31:40.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:31:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:31:40.626+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:31:40.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:31:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:31:40.626+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:31:40.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:31:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:31:40.627+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:31:40.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:31:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:31:40.628+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:31:40.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:31:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:31:40.628+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:31:40.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:31:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:31:40.628+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:31:40.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:31:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:31:40.628+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:31:40.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:31:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:31:40.522+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:31:40.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:31:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:31:40.522+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:31:40.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:31:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:31:40.522+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:31:40.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:31:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:31:40.522+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:31:42.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:31:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:31:41.570+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:31:42.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:31:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:31:41.570+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:31:42.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:31:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:31:41.570+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:31:42.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:31:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:31:41.571+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:31:42.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:31:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:31:41.593+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:31:42.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:31:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:31:41.593+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:31:42.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:31:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:31:41.593+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:31:42.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:31:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:31:41.593+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:31:42.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:31:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:31:41.600+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:31:42.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:31:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:31:41.600+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:31:42.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:31:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:31:41.600+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:31:42.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:31:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:31:41.601+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:31:43.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:31:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:31:42.574+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:31:43.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:31:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:31:42.574+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:31:43.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:31:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:31:42.574+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:31:43.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:31:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:31:42.574+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:31:43.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:31:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:31:42.642+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:31:43.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:31:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:31:42.642+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:31:43.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:31:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:31:42.642+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:31:43.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:31:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:31:42.643+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:31:43.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:31:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:31:42.609+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:31:43.071 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:31:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:31:42.609+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:31:43.071 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:31:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:31:42.609+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:31:43.071 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:31:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:31:42.609+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:31:43.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:31:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:31:43.615+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:31:43.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:31:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:31:43.615+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:31:43.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:31:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:31:43.615+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:31:43.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:31:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:31:43.615+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:31:43.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:31:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:31:43.628+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:31:43.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:31:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:31:43.628+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:31:43.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:31:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:31:43.628+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:31:43.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:31:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:31:43.629+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:31:43.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:31:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:31:43.567+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:31:43.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:31:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:31:43.567+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:31:43.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:31:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:31:43.567+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:31:43.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:31:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:31:43.568+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:31:44.795 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:31:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:31:44.592+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:31:44.795 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:31:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:31:44.592+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:31:44.795 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:31:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:31:44.592+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:31:44.796 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:31:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:31:44.592+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:31:44.796 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:31:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:31:44.650+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:31:44.796 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:31:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:31:44.650+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:31:44.796 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:31:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:31:44.650+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:31:44.796 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:31:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:31:44.651+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:31:44.797 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:31:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:31:44.533+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:31:44.797 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:31:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:31:44.533+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:31:44.797 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:31:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:31:44.533+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:31:44.797 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:31:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:31:44.533+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:31:45.492 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:31:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:31:45.228+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 8516 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:31:45.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:31:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:31:45.492+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:31:45.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:31:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:31:45.492+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:31:45.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:31:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:31:45.492+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:31:45.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:31:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:31:45.492+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:31:45.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:31:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:31:45.618+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:31:45.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:31:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:31:45.618+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:31:45.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:31:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:31:45.618+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:31:45.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:31:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:31:45.619+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:31:45.821 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:31:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:31:45.621+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:31:45.821 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:31:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:31:45.621+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:31:45.821 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:31:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:31:45.621+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:31:45.821 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:31:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:31:45.622+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:31:46.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:31:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:31:46.483+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:31:46.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:31:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:31:46.483+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:31:46.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:31:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:31:46.483+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:31:46.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:31:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:31:46.483+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:31:46.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:31:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:31:46.664+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:31:46.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:31:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:31:46.664+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:31:46.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:31:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:31:46.664+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:31:46.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:31:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:31:46.664+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:31:46.821 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:31:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:31:46.618+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:31:46.821 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:31:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:31:46.618+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:31:46.821 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:31:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:31:46.618+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:31:46.821 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:31:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:31:46.619+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:31:47.699 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:31:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:31:47.624+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:31:47.699 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:31:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:31:47.624+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:31:47.700 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:31:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:31:47.624+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:31:47.700 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:31:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:31:47.625+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:31:47.700 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:31:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:31:47.434+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:31:47.700 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:31:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:31:47.434+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:31:47.700 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:31:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:31:47.434+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:31:47.700 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:31:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:31:47.435+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:31:48.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:31:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:31:47.698+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:31:48.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:31:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:31:47.698+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:31:48.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:31:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:31:47.698+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:31:48.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:31:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:31:47.699+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:31:48.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:31:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:31:48.679+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:31:48.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:31:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:31:48.679+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:31:48.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:31:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:31:48.679+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:31:48.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:31:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:31:48.679+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:31:48.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:31:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:31:48.582+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:31:48.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:31:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:31:48.582+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:31:48.821 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:31:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:31:48.582+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:31:48.821 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:31:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:31:48.582+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:31:48.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:31:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:31:48.434+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:31:48.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:31:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:31:48.434+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:31:48.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:31:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:31:48.434+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:31:48.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:31:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:31:48.435+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:31:49.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:31:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:31:49.472+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:31:49.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:31:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:31:49.472+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:31:49.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:31:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:31:49.472+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:31:49.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:31:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:31:49.472+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:31:49.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:31:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:31:49.648+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:31:49.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:31:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:31:49.648+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:31:49.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:31:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:31:49.648+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:31:49.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:31:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:31:49.648+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:31:49.821 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:31:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:31:49.604+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:31:49.821 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:31:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:31:49.604+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:31:49.821 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:31:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:31:49.604+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:31:49.821 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:31:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:31:49.605+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:31:50.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:31:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:31:50.229+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 8609 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:31:50.569 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:31:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:31:50.435+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:31:50.569 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:31:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:31:50.435+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:31:50.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:31:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:31:50.435+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:31:50.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:31:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:31:50.435+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:31:51.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:31:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:31:50.626+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:31:51.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:31:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:31:50.626+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:31:51.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:31:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:31:50.626+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:31:51.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:31:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:31:50.626+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:31:51.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:31:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:31:50.635+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:31:51.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:31:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:31:50.635+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:31:51.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:31:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:31:50.635+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:31:51.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:31:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:31:50.636+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:31:51.668 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:31:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:31:51.583+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:31:51.669 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:31:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:31:51.583+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:31:51.669 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:31:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:31:51.583+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:31:51.669 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:31:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:31:51.584+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:31:51.669 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:31:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:31:51.630+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:31:51.669 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:31:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:31:51.630+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:31:51.669 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:31:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:31:51.630+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:31:51.670 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:31:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:31:51.631+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:31:51.670 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:31:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:31:51.408+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:31:51.670 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:31:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:31:51.408+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:31:51.670 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:31:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:31:51.408+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:31:51.670 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:31:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:31:51.408+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:31:52.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:31:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:31:52.452+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:31:52.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:31:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:31:52.452+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:31:52.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:31:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:31:52.452+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:31:52.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:31:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:31:52.453+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:31:52.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:31:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:31:52.596+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:31:52.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:31:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:31:52.596+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:31:52.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:31:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:31:52.596+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:31:52.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:31:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:31:52.597+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:31:52.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:31:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:31:52.558+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:31:52.821 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:31:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:31:52.558+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:31:52.821 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:31:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:31:52.558+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:31:52.821 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:31:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:31:52.559+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:31:53.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:31:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:31:53.557+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:31:53.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:31:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:31:53.557+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:31:53.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:31:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:31:53.557+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:31:53.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:31:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:31:53.558+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:31:53.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:31:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:31:53.494+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:31:53.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:31:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:31:53.494+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:31:53.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:31:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:31:53.494+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:31:53.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:31:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:31:53.494+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:31:53.821 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:31:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:31:53.633+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:31:53.821 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:31:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:31:53.633+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:31:53.821 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:31:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:31:53.633+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:31:53.821 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:31:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:31:53.633+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:31:54.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:31:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:31:54.484+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:31:54.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:31:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:31:54.484+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:31:54.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:31:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:31:54.484+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:31:54.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:31:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:31:54.484+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:31:54.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:31:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:31:54.627+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:31:54.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:31:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:31:54.627+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:31:54.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:31:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:31:54.627+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:31:54.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:31:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:31:54.628+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:31:54.821 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:31:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:31:54.530+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:31:54.821 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:31:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:31:54.530+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:31:54.821 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:31:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:31:54.530+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:31:54.821 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:31:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:31:54.531+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:31:55.529 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:31:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:31:55.229+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 8703 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:31:55.529 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:31:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:31:55.476+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:31:55.530 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:31:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:31:55.476+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:31:55.530 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:31:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:31:55.476+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:31:55.530 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:31:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:31:55.476+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:31:55.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:31:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:31:55.613+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:31:55.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:31:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:31:55.613+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:31:55.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:31:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:31:55.613+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:31:55.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:31:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:31:55.614+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:31:55.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:31:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:31:55.527+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:31:55.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:31:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:31:55.527+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:31:55.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:31:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:31:55.527+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:31:55.821 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:31:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:31:55.528+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:31:56.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:31:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:31:56.632+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:31:56.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:31:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:31:56.632+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:31:56.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:31:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:31:56.632+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:31:56.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:31:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:31:56.632+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:31:56.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:31:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:31:56.554+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:31:56.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:31:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:31:56.554+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:31:56.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:31:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:31:56.554+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:31:56.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:31:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:31:56.554+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:31:56.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:31:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:31:56.458+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:31:56.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:31:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:31:56.458+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:31:56.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:31:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:31:56.458+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:31:56.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:31:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:31:56.459+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:31:57.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:31:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:31:57.521+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:31:57.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:31:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:31:57.521+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:31:57.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:31:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:31:57.521+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:31:57.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:31:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:31:57.522+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:31:57.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:31:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:31:57.676+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:31:57.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:31:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:31:57.676+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:31:57.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:31:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:31:57.676+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:31:57.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:31:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:31:57.677+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:31:57.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:31:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:31:57.480+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:31:57.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:31:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:31:57.480+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:31:57.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:31:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:31:57.480+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:31:57.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:31:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:31:57.480+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:31:58.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:31:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:31:58.501+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:31:58.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:31:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:31:58.501+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:31:58.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:31:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:31:58.501+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:31:58.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:31:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:31:58.501+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:31:58.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:31:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:31:58.697+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:31:58.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:31:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:31:58.697+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:31:58.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:31:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:31:58.697+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:31:58.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:31:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:31:58.697+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:31:58.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:31:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:31:58.466+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:31:58.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:31:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:31:58.466+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:31:58.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:31:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:31:58.466+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:31:58.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:31:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:31:58.467+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:31:59.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:31:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:31:59.498+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:31:59.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:31:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:31:59.498+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:31:59.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:31:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:31:59.498+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:31:59.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:31:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:31:59.498+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:31:59.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:31:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:31:59.683+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:31:59.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:31:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:31:59.683+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:31:59.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:31:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:31:59.683+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:31:59.821 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:31:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:31:59.683+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:31:59.821 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:31:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:31:59.525+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:31:59.821 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:31:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:31:59.525+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:31:59.821 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:31:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:31:59.525+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:31:59.821 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:31:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:31:59.525+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:32:00.513 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:32:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:32:00.231+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 8795 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:32:00.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:32:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:32:00.519+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:32:00.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:32:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:32:00.519+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:32:00.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:32:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:32:00.519+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:32:00.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:32:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:32:00.520+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:32:00.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:32:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:32:00.652+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:32:00.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:32:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:32:00.652+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:32:00.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:32:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:32:00.652+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:32:00.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:32:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:32:00.652+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:32:00.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:32:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:32:00.512+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:32:00.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:32:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:32:00.512+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:32:00.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:32:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:32:00.512+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:32:00.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:32:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:32:00.512+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:32:01.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:32:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:32:01.677+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:32:01.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:32:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:32:01.677+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:32:01.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:32:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:32:01.677+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:32:01.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:32:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:32:01.678+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:32:01.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:32:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:32:01.538+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:32:01.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:32:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:32:01.538+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:32:01.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:32:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:32:01.538+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:32:01.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:32:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:32:01.538+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:32:01.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:32:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:32:01.482+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:32:01.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:32:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:32:01.482+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:32:01.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:32:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:32:01.482+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:32:01.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:32:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:32:01.482+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:32:02.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:32:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:32:02.706+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:32:02.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:32:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:32:02.706+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:32:02.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:32:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:32:02.706+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:32:02.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:32:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:32:02.706+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:32:02.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:32:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:32:02.515+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:32:02.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:32:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:32:02.515+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:32:02.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:32:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:32:02.515+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:32:02.822 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:32:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:32:02.515+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:32:02.822 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:32:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:32:02.576+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:32:02.822 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:32:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:32:02.576+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:32:02.822 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:32:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:32:02.576+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:32:02.822 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:32:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:32:02.577+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:32:03.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:32:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:32:03.564+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:32:03.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:32:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:32:03.564+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:32:03.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:32:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:32:03.564+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:32:03.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:32:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:32:03.565+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:32:03.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:32:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:32:03.731+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:32:03.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:32:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:32:03.731+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:32:03.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:32:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:32:03.731+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:32:03.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:32:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:32:03.731+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:32:03.821 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:32:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:32:03.559+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:32:03.821 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:32:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:32:03.559+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:32:03.821 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:32:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:32:03.559+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:32:03.821 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:32:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:32:03.559+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:32:04.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:32:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:32:04.535+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:32:04.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:32:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:32:04.535+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:32:04.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:32:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:32:04.535+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:32:04.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:32:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:32:04.535+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:32:04.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:32:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:32:04.780+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:32:04.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:32:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:32:04.780+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:32:04.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:32:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:32:04.780+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:32:04.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:32:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:32:04.781+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:32:04.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:32:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:32:04.592+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:32:04.821 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:32:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:32:04.592+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:32:04.821 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:32:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:32:04.592+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:32:04.821 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:32:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:32:04.593+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:32:05.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:32:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:32:05.233+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 8887 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:32:06.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:32:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:32:05.582+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:32:06.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:32:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:32:05.582+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:32:06.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:32:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:32:05.582+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:32:06.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:32:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:32:05.582+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:32:06.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:32:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:32:05.734+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:32:06.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:32:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:32:05.734+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:32:06.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:32:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:32:05.734+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:32:06.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:32:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:32:05.735+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:32:06.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:32:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:32:05.575+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:32:06.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:32:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:32:05.575+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:32:06.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:32:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:32:05.575+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:32:06.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:32:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:32:05.576+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:32:07.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:32:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:32:06.617+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:32:07.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:32:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:32:06.617+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:32:07.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:32:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:32:06.617+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:32:07.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:32:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:32:06.617+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:32:07.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:32:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:32:06.714+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:32:07.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:32:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:32:06.715+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:32:07.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:32:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:32:06.715+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:32:07.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:32:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:32:06.716+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:32:07.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:32:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:32:06.578+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:32:07.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:32:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:32:06.578+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:32:07.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:32:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:32:06.578+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:32:07.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:32:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:32:06.579+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:32:07.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:32:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:32:07.657+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:32:07.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:32:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:32:07.657+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:32:07.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:32:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:32:07.657+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:32:07.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:32:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:32:07.657+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:32:07.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:32:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:32:07.675+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:32:07.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:32:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:32:07.675+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:32:07.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:32:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:32:07.675+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:32:07.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:32:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:32:07.675+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:32:07.821 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:32:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:32:07.535+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:32:07.821 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:32:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:32:07.535+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:32:07.821 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:32:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:32:07.535+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:32:07.821 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:32:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:32:07.536+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:32:09.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:32:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:32:08.585+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:32:09.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:32:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:32:08.585+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:32:09.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:32:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:32:08.585+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:32:09.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:32:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:32:08.586+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:32:09.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:32:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:32:08.660+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:32:09.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:32:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:32:08.660+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:32:09.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:32:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:32:08.660+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:32:09.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:32:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:32:08.661+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:32:09.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:32:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:32:08.672+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:32:09.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:32:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:32:08.672+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:32:09.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:32:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:32:08.672+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:32:09.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:32:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:32:08.672+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:32:10.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:32:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:32:09.693+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:32:10.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:32:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:32:09.693+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:32:10.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:32:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:32:09.693+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:32:10.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:32:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:32:09.693+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:32:10.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:32:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:32:09.643+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:32:10.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:32:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:32:09.643+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:32:10.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:32:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:32:09.643+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:32:10.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:32:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:32:09.643+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:32:10.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:32:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:32:09.583+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:32:10.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:32:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:32:09.583+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:32:10.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:32:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:32:09.583+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:32:10.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:32:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:32:09.583+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:32:10.567 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:32:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:32:10.234+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 8977 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:32:10.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:32:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:32:10.651+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:32:10.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:32:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:32:10.651+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:32:10.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:32:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:32:10.651+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:32:10.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:32:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:32:10.651+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:32:10.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:32:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:32:10.566+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:32:10.821 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:32:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:32:10.566+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:32:10.821 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:32:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:32:10.566+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:32:10.821 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:32:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:32:10.567+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:32:10.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:32:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:32:10.718+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:32:10.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:32:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:32:10.718+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:32:10.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:32:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:32:10.718+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:32:10.822 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:32:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:32:10.719+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:32:11.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:32:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:32:11.610+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:32:11.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:32:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:32:11.610+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:32:11.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:32:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:32:11.610+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:32:11.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:32:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:32:11.611+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:32:11.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:32:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:32:11.747+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:32:11.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:32:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:32:11.747+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:32:11.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:32:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:32:11.747+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:32:11.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:32:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:32:11.747+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:32:11.821 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:32:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:32:11.536+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:32:11.821 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:32:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:32:11.536+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:32:11.821 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:32:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:32:11.536+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:32:11.821 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:32:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:32:11.537+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:32:12.584 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-18T01:32:12.584 WARNING:tasks.kclient:Ordinary umount failed, forcing... 2024-03-18T01:32:12.585 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-03-18T01:32:12.585 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-18T01:32:12.602 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-03-18T01:32:12.602 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-18T01:32:12.659 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T01:32:12.720 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:32:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:32:12.508+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:32:12.721 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:32:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:32:12.508+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:32:12.721 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:32:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:32:12.508+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:32:12.721 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:32:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:32:12.509+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:32:12.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:32:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:32:12.565+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:32:12.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:32:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:32:12.565+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:32:12.722 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:32:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:32:12.565+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:32:12.722 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:32:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:32:12.565+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:32:13.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:32:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:32:12.720+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:32:13.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:32:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:32:12.720+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:32:13.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:32:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:32:12.720+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:32:13.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:32:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:32:12.721+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:32:13.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:32:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:32:13.478+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:32:13.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:32:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:32:13.478+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:32:13.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:32:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:32:13.478+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:32:13.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:32:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:32:13.479+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:32:13.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:32:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:32:13.691+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:32:13.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:32:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:32:13.691+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:32:13.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:32:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:32:13.691+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:32:13.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:32:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:32:13.692+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:32:13.821 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:32:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:32:13.575+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:32:13.821 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:32:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:32:13.575+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:32:13.821 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:32:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:32:13.575+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:32:13.821 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:32:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:32:13.576+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:32:14.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:32:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:32:14.521+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:32:14.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:32:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:32:14.521+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:32:14.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:32:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:32:14.521+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:32:14.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:32:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:32:14.521+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:32:14.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:32:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:32:14.602+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:32:14.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:32:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:32:14.602+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:32:14.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:32:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:32:14.602+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:32:14.821 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:32:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:32:14.602+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:32:14.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:32:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:32:14.666+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:32:14.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:32:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:32:14.666+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:32:14.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:32:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:32:14.666+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:32:14.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:32:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:32:14.666+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:32:15.500 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:32:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:32:15.235+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 9065 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:32:15.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:32:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:32:15.629+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:32:15.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:32:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:32:15.629+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:32:15.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:32:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:32:15.629+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:32:15.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:32:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:32:15.630+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:32:15.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:32:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:32:15.499+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:32:15.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:32:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:32:15.499+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:32:15.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:32:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:32:15.499+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:32:15.821 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:32:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:32:15.500+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:32:15.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:32:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:32:15.677+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:32:15.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:32:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:32:15.677+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:32:15.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:32:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:32:15.677+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:32:15.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:32:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:32:15.677+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:32:16.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:32:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:32:16.634+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:32:16.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:32:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:32:16.634+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:32:16.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:32:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:32:16.634+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:32:16.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:32:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:32:16.635+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:32:16.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:32:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:32:16.702+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:32:16.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:32:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:32:16.702+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:32:16.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:32:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:32:16.702+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:32:16.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:32:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:32:16.703+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:32:16.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:32:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:32:16.522+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:32:16.821 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:32:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:32:16.522+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:32:16.821 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:32:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:32:16.522+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:32:16.821 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:32:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:32:16.523+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:32:17.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:32:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:32:17.638+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:32:17.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:32:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:32:17.638+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:32:17.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:32:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:32:17.638+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:32:17.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:32:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:32:17.639+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:32:17.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:32:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:32:17.525+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:32:17.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:32:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:32:17.525+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:32:17.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:32:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:32:17.525+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:32:17.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:32:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:32:17.525+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:32:17.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:32:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:32:17.705+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:32:17.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:32:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:32:17.705+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:32:17.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:32:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:32:17.705+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:32:17.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:32:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:32:17.706+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:32:18.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:32:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:32:18.657+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:32:18.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:32:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:32:18.657+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:32:18.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:32:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:32:18.657+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:32:18.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:32:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:32:18.658+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:32:18.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:32:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:32:18.528+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:32:18.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:32:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:32:18.528+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:32:18.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:32:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:32:18.528+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:32:18.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:32:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:32:18.529+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:32:18.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:32:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:32:18.701+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:32:18.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:32:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:32:18.701+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:32:18.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:32:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:32:18.701+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:32:18.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:32:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:32:18.702+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:32:19.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:32:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:32:19.511+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:32:19.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:32:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:32:19.511+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:32:19.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:32:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:32:19.511+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:32:19.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:32:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:32:19.512+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:32:19.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:32:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:32:19.735+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:32:19.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:32:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:32:19.735+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:32:19.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:32:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:32:19.735+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:32:19.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:32:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:32:19.735+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:32:19.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:32:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:32:19.614+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:32:19.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:32:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:32:19.614+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:32:19.821 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:32:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:32:19.614+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:32:19.821 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:32:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:32:19.615+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:32:20.491 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:32:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:32:20.236+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 9148 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:32:20.744 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:32:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:32:20.650+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:32:20.744 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:32:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:32:20.650+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:32:20.744 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:32:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:32:20.650+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:32:20.744 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:32:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:32:20.651+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:32:20.744 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:32:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:32:20.491+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:32:20.745 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:32:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:32:20.491+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:32:20.745 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:32:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:32:20.491+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:32:20.745 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:32:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:32:20.492+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:32:21.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:32:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:32:20.743+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:32:21.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:32:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:32:20.743+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:32:21.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:32:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:32:20.743+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:32:21.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:32:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:32:20.744+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:32:21.775 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:32:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:32:21.471+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:32:21.775 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:32:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:32:21.471+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:32:21.775 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:32:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:32:21.471+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:32:21.775 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:32:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:32:21.472+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:32:21.775 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:32:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:32:21.602+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:32:21.776 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:32:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:32:21.602+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:32:21.776 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:32:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:32:21.602+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:32:21.776 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:32:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:32:21.603+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:32:22.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:32:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:32:21.774+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:32:22.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:32:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:32:21.774+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:32:22.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:32:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:32:21.774+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:32:22.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:32:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:32:21.774+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:32:22.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:32:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:32:22.443+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:32:22.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:32:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:32:22.443+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:32:22.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:32:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:32:22.443+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:32:22.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:32:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:32:22.444+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:32:22.731 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:32:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:32:22.555+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:32:22.731 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:32:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:32:22.555+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:32:22.731 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:32:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:32:22.555+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:32:22.731 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:32:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:32:22.555+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:32:23.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:32:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:32:22.729+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:32:23.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:32:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:32:22.729+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:32:23.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:32:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:32:22.729+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:32:23.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:32:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:32:22.729+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:32:23.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:32:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:32:23.685+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:32:23.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:32:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:32:23.685+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:32:23.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:32:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:32:23.685+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:32:23.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:32:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:32:23.685+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:32:23.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:32:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:32:23.534+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:32:23.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:32:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:32:23.534+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:32:23.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:32:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:32:23.534+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:32:23.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:32:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:32:23.535+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:32:23.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:32:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:32:23.462+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:32:23.821 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:32:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:32:23.462+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:32:23.821 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:32:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:32:23.462+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:32:23.821 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:32:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:32:23.463+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:32:24.731 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:32:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:32:24.465+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:32:24.731 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:32:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:32:24.465+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:32:24.731 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:32:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:32:24.465+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:32:24.731 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:32:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:32:24.465+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:32:24.732 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:32:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:32:24.701+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:32:24.732 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:32:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:32:24.701+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:32:24.732 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:32:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:32:24.701+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:32:24.732 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:32:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:32:24.702+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:32:24.732 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:32:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:32:24.537+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:32:24.732 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:32:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:32:24.537+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:32:24.733 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:32:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:32:24.537+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:32:24.733 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:32:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:32:24.538+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:32:25.505 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:32:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:32:25.237+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 9234 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:32:25.506 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:32:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:32:25.455+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:32:25.506 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:32:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:32:25.455+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:32:25.506 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:32:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:32:25.455+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:32:25.506 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:32:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:32:25.456+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:32:25.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:32:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:32:25.505+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:32:25.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:32:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:32:25.505+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:32:25.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:32:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:32:25.505+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:32:25.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:32:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:32:25.506+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:32:25.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:32:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:32:25.723+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:32:25.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:32:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:32:25.723+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:32:25.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:32:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:32:25.723+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:32:25.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:32:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:32:25.724+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:32:26.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:32:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:32:26.464+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:32:26.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:32:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:32:26.464+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:32:26.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:32:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:32:26.464+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:32:26.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:32:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:32:26.465+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:32:26.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:32:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:32:26.695+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:32:26.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:32:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:32:26.695+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:32:26.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:32:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:32:26.695+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:32:26.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:32:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:32:26.695+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:32:26.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:32:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:32:26.454+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:32:26.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:32:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:32:26.454+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:32:26.821 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:32:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:32:26.454+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:32:26.821 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:32:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:32:26.455+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:32:27.803 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:32:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:32:27.486+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:32:27.804 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:32:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:32:27.486+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:32:27.804 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:32:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:32:27.486+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:32:27.804 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:32:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:32:27.487+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:32:27.804 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:32:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:32:27.664+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:32:27.805 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:32:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:32:27.664+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:32:27.805 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:32:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:32:27.664+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:32:27.805 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:32:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:32:27.664+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:32:27.805 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:32:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:32:27.509+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:32:27.805 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:32:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:32:27.509+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:32:27.805 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:32:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:32:27.509+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:32:27.806 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:32:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:32:27.509+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:32:28.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:32:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:32:28.697+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:32:28.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:32:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:32:28.697+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:32:28.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:32:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:32:28.697+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:32:28.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:32:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:32:28.697+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:32:28.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:32:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:32:28.498+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:32:28.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:32:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:32:28.498+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:32:28.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:32:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:32:28.498+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:32:28.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:32:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:32:28.499+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:32:28.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:32:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:32:28.479+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:32:28.821 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:32:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:32:28.479+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:32:28.821 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:32:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:32:28.479+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:32:28.821 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:32:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:32:28.480+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:32:29.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:32:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:32:29.545+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:32:29.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:32:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:32:29.545+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:32:29.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:32:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:32:29.545+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:32:29.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:32:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:32:29.546+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:32:29.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:32:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:32:29.487+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:32:29.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:32:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:32:29.487+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:32:29.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:32:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:32:29.487+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:32:29.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:32:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:32:29.488+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:32:29.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:32:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:32:29.704+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:32:29.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:32:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:32:29.704+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:32:29.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:32:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:32:29.704+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:32:29.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:32:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:32:29.705+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:32:30.492 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:32:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:32:30.238+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 9321 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:32:30.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:32:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:32:30.672+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:32:30.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:32:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:32:30.672+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:32:30.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:32:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:32:30.672+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:32:30.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:32:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:32:30.673+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:32:30.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:32:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:32:30.491+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:32:30.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:32:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:32:30.491+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:32:30.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:32:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:32:30.491+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:32:30.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:32:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:32:30.492+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:32:30.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:32:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:32:30.539+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:32:30.821 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:32:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:32:30.539+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:32:30.821 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:32:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:32:30.539+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:32:30.821 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:32:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:32:30.540+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:32:31.822 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:32:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:32:31.522+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:32:31.822 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:32:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:32:31.522+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:32:31.822 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:32:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:32:31.522+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:32:31.822 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:32:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:32:31.523+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:32:31.823 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:32:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:32:31.531+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:32:31.823 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:32:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:32:31.531+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:32:31.823 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:32:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:32:31.531+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:32:31.823 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:32:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:32:31.532+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:32:31.823 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:32:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:32:31.624+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:32:31.824 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:32:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:32:31.624+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:32:31.824 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:32:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:32:31.624+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:32:31.824 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:32:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:32:31.624+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:32:32.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:32:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:32:32.484+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:32:32.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:32:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:32:32.484+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:32:32.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:32:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:32:32.484+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:32:32.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:32:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:32:32.484+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:32:32.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:32:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:32:32.548+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:32:32.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:32:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:32:32.548+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:32:32.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:32:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:32:32.548+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:32:32.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:32:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:32:32.548+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:32:32.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:32:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:32:32.582+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:32:32.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:32:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:32:32.582+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:32:32.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:32:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:32:32.582+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:32:32.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:32:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:32:32.583+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:32:33.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:32:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:32:33.498+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:32:33.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:32:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:32:33.498+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:32:33.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:32:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:32:33.498+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:32:33.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:32:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:32:33.499+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:32:33.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:32:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:32:33.572+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:32:33.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:32:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:32:33.572+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:32:33.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:32:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:32:33.572+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:32:33.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:32:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:32:33.573+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:32:33.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:32:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:32:33.584+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:32:33.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:32:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:32:33.584+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:32:33.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:32:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:32:33.584+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:32:33.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:32:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:32:33.585+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:32:34.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:32:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:32:34.460+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:32:34.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:32:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:32:34.460+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:32:34.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:32:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:32:34.460+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:32:34.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:32:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:32:34.460+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:32:34.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:32:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:32:34.608+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:32:34.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:32:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:32:34.608+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:32:34.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:32:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:32:34.608+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:32:34.821 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:32:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:32:34.608+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:32:34.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:32:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:32:34.541+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:32:34.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:32:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:32:34.541+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:32:34.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:32:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:32:34.541+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:32:34.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:32:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:32:34.541+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:32:35.565 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:32:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:32:35.239+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 9409 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:32:35.565 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:32:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:32:35.446+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:32:35.565 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:32:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:32:35.446+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:32:35.565 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:32:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:32:35.446+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:32:35.566 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:32:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:32:35.447+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:32:35.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:32:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:32:35.564+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:32:35.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:32:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:32:35.564+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:32:35.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:32:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:32:35.564+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:32:35.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:32:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:32:35.565+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:32:35.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:32:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:32:35.611+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:32:35.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:32:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:32:35.611+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:32:35.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:32:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:32:35.611+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:32:35.821 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:32:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:32:35.611+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:32:36.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:32:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:32:36.636+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:32:36.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:32:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:32:36.636+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:32:36.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:32:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:32:36.636+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:32:36.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:32:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:32:36.638+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:32:36.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:32:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:32:36.601+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:32:36.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:32:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:32:36.601+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:32:36.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:32:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:32:36.601+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:32:36.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:32:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:32:36.601+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:32:36.821 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:32:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:32:36.466+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:32:36.821 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:32:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:32:36.466+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:32:36.821 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:32:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:32:36.466+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:32:36.821 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:32:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:32:36.466+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:32:37.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:32:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:32:37.482+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:32:37.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:32:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:32:37.482+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:32:37.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:32:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:32:37.482+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:32:37.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:32:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:32:37.484+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:32:37.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:32:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:32:37.588+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:32:37.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:32:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:32:37.588+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:32:37.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:32:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:32:37.588+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:32:37.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:32:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:32:37.588+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:32:37.821 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:32:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:32:37.599+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:32:37.821 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:32:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:32:37.599+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:32:37.821 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:32:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:32:37.599+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:32:37.821 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:32:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:32:37.600+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:32:38.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:32:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:32:38.557+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:32:38.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:32:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:32:38.557+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:32:38.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:32:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:32:38.557+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:32:38.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:32:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:32:38.557+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:32:38.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:32:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:32:38.525+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:32:38.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:32:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:32:38.525+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:32:38.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:32:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:32:38.525+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:32:38.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:32:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:32:38.526+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:32:38.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:32:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:32:38.566+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:32:38.821 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:32:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:32:38.566+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:32:38.821 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:32:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:32:38.566+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:32:38.821 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:32:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:32:38.567+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:32:39.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:32:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:32:39.555+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:32:39.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:32:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:32:39.555+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:32:39.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:32:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:32:39.555+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:32:39.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:32:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:32:39.555+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:32:39.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:32:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:32:39.597+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:32:39.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:32:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:32:39.597+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:32:39.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:32:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:32:39.597+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:32:39.821 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:32:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:32:39.598+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:32:39.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:32:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:32:39.514+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:32:39.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:32:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:32:39.514+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:32:39.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:32:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:32:39.514+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:32:39.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:32:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:32:39.515+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:32:40.507 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:32:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:32:40.241+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 9499 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:32:40.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:32:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:32:40.606+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:32:40.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:32:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:32:40.606+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:32:40.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:32:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:32:40.606+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:32:40.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:32:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:32:40.607+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:32:40.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:32:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:32:40.544+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:32:40.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:32:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:32:40.544+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:32:40.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:32:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:32:40.544+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:32:40.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:32:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:32:40.545+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:32:40.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:32:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:32:40.506+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:32:40.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:32:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:32:40.506+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:32:40.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:32:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:32:40.506+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:32:40.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:32:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:32:40.507+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:32:41.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:32:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:32:41.648+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:32:41.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:32:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:32:41.648+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:32:41.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:32:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:32:41.648+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:32:41.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:32:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:32:41.648+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:32:41.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:32:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:32:41.495+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:32:41.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:32:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:32:41.495+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:32:41.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:32:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:32:41.495+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:32:41.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:32:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:32:41.495+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:32:41.821 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:32:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:32:41.592+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:32:41.822 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:32:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:32:41.592+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:32:41.822 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:32:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:32:41.592+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:32:41.822 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:32:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:32:41.593+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:32:42.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:32:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:32:42.687+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:32:42.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:32:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:32:42.687+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:32:42.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:32:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:32:42.687+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:32:42.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:32:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:32:42.687+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:32:42.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:32:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:32:42.622+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:32:42.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:32:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:32:42.622+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:32:42.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:32:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:32:42.622+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:32:42.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:32:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:32:42.622+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:32:42.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:32:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:32:42.535+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:32:42.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:32:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:32:42.535+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:32:42.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:32:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:32:42.535+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:32:42.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:32:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:32:42.536+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:32:43.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:32:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:32:43.664+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:32:43.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:32:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:32:43.664+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:32:43.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:32:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:32:43.664+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:32:43.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:32:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:32:43.665+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:32:43.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:32:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:32:43.647+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:32:43.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:32:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:32:43.647+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:32:43.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:32:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:32:43.647+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:32:43.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:32:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:32:43.647+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:32:43.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:32:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:32:43.584+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:32:43.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:32:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:32:43.584+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:32:43.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:32:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:32:43.584+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:32:43.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:32:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:32:43.584+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:32:45.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:32:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:32:44.613+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:32:45.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:32:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:32:44.613+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:32:45.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:32:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:32:44.613+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:32:45.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:32:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:32:44.614+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:32:45.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:32:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:32:44.609+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:32:45.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:32:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:32:44.609+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:32:45.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:32:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:32:44.609+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:32:45.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:32:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:32:44.609+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:32:45.071 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:32:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:32:44.682+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:32:45.071 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:32:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:32:44.682+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:32:45.071 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:32:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:32:44.682+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:32:45.071 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:32:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:32:44.682+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:32:45.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:32:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:32:45.242+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 9593 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:32:46.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:32:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:32:45.676+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:32:46.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:32:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:32:45.676+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:32:46.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:32:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:32:45.676+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:32:46.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:32:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:32:45.676+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:32:46.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:32:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:32:45.625+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:32:46.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:32:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:32:45.625+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:32:46.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:32:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:32:45.625+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:32:46.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:32:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:32:45.626+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:32:46.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:32:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:32:45.645+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:32:46.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:32:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:32:45.645+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:32:46.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:32:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:32:45.645+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:32:46.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:32:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:32:45.645+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:32:47.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:32:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:32:46.663+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:32:47.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:32:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:32:46.663+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:32:47.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:32:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:32:46.663+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:32:47.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:32:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:32:46.664+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:32:47.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:32:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:32:46.606+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:32:47.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:32:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:32:46.606+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:32:47.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:32:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:32:46.606+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:32:47.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:32:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:32:46.606+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:32:47.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:32:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:32:46.590+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:32:47.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:32:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:32:46.590+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:32:47.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:32:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:32:46.590+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:32:47.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:32:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:32:46.591+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:32:48.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:32:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:32:47.650+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:32:48.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:32:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:32:47.650+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:32:48.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:32:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:32:47.650+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:32:48.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:32:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:32:47.651+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:32:48.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:32:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:32:47.584+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:32:48.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:32:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:32:47.584+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:32:48.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:32:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:32:47.584+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:32:48.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:32:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:32:47.585+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:32:48.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:32:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:32:47.646+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:32:48.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:32:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:32:47.646+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:32:48.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:32:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:32:47.646+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:32:48.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:32:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:32:47.647+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:32:49.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:32:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:32:48.665+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:32:49.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:32:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:32:48.665+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:32:49.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:32:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:32:48.665+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:32:49.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:32:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:32:48.665+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:32:49.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:32:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:32:48.571+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:32:49.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:32:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:32:48.571+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:32:49.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:32:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:32:48.571+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:32:49.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:32:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:32:48.572+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:32:49.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:32:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:32:48.643+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:32:49.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:32:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:32:48.643+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:32:49.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:32:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:32:48.643+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:32:49.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:32:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:32:48.643+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:32:49.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:32:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:32:49.663+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:32:49.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:32:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:32:49.663+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:32:49.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:32:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:32:49.663+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:32:49.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:32:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:32:49.664+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:32:49.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:32:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:32:49.526+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:32:49.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:32:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:32:49.526+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:32:49.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:32:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:32:49.526+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:32:49.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:32:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:32:49.527+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:32:49.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:32:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:32:49.601+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:32:49.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:32:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:32:49.601+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:32:49.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:32:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:32:49.601+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:32:49.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:32:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:32:49.601+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:32:50.494 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:32:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:32:50.243+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 9687 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:32:50.494 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:32:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:32:50.493+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:32:50.495 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:32:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:32:50.493+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:32:50.495 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:32:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:32:50.493+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:32:50.495 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:32:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:32:50.493+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:32:50.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:32:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:32:50.638+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:32:50.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:32:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:32:50.638+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:32:50.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:32:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:32:50.638+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:32:50.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:32:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:32:50.639+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:32:50.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:32:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:32:50.601+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:32:50.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:32:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:32:50.601+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:32:50.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:32:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:32:50.601+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:32:50.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:32:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:32:50.601+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:32:51.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:32:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:32:51.616+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:32:51.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:32:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:32:51.616+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:32:51.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:32:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:32:51.616+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:32:51.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:32:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:32:51.616+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:32:51.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:32:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:32:51.616+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:32:51.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:32:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:32:51.616+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:32:51.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:32:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:32:51.616+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:32:51.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:32:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:32:51.617+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:32:51.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:32:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:32:51.448+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:32:51.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:32:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:32:51.448+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:32:51.821 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:32:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:32:51.448+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:32:51.821 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:32:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:32:51.449+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:32:52.634 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:32:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:32:52.425+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:32:52.635 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:32:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:32:52.425+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:32:52.635 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:32:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:32:52.425+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:32:52.635 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:32:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:32:52.426+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:32:52.635 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:32:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:32:52.588+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:32:52.635 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:32:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:32:52.588+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:32:52.635 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:32:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:32:52.588+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:32:52.636 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:32:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:32:52.589+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:32:53.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:32:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:32:52.634+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:32:53.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:32:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:32:52.634+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:32:53.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:32:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:32:52.634+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:32:53.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:32:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:32:52.635+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:32:53.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:32:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:32:53.436+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:32:53.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:32:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:32:53.436+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:32:53.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:32:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:32:53.436+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:32:53.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:32:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:32:53.436+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:32:53.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:32:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:32:53.683+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:32:53.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:32:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:32:53.683+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:32:53.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:32:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:32:53.683+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:32:53.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:32:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:32:53.683+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:32:53.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:32:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:32:53.604+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:32:53.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:32:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:32:53.604+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:32:53.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:32:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:32:53.604+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:32:53.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:32:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:32:53.605+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:32:54.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:32:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:32:54.414+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:32:54.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:32:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:32:54.414+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:32:54.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:32:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:32:54.414+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:32:54.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:32:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:32:54.415+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:32:54.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:32:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:32:54.596+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:32:54.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:32:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:32:54.596+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:32:54.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:32:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:32:54.596+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:32:54.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:32:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:32:54.596+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:32:55.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:32:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:32:54.725+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:32:55.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:32:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:32:54.725+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:32:55.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:32:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:32:54.725+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:32:55.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:32:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:32:54.725+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:32:55.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:32:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:32:55.243+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 9781 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:32:55.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:32:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:32:55.430+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:32:55.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:32:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:32:55.430+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:32:55.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:32:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:32:55.430+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:32:55.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:32:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:32:55.431+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:32:56.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:32:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:32:55.715+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:32:56.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:32:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:32:55.715+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:32:56.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:32:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:32:55.715+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:32:56.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:32:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:32:55.716+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:32:56.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:32:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:32:55.637+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:32:56.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:32:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:32:55.637+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:32:56.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:32:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:32:55.637+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:32:56.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:32:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:32:55.638+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:32:56.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:32:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:32:56.700+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:32:56.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:32:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:32:56.700+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:32:56.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:32:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:32:56.700+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:32:56.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:32:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:32:56.701+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:32:56.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:32:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:32:56.452+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:32:56.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:32:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:32:56.452+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:32:56.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:32:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:32:56.452+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:32:56.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:32:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:32:56.453+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:32:56.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:32:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:32:56.590+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:32:56.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:32:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:32:56.590+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:32:56.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:32:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:32:56.590+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:32:56.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:32:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:32:56.591+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:32:57.724 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:32:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:32:57.457+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:32:57.724 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:32:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:32:57.457+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:32:57.724 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:32:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:32:57.458+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:32:57.725 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:32:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:32:57.459+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:32:57.725 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:32:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:32:57.567+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:32:57.725 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:32:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:32:57.567+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:32:57.725 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:32:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:32:57.567+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:32:57.725 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:32:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:32:57.568+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:32:58.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:32:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:32:57.722+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:32:58.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:32:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:32:57.722+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:32:58.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:32:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:32:57.722+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:32:58.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:32:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:32:57.723+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:32:58.775 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:32:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:32:58.610+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:32:58.775 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:32:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:32:58.610+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:32:58.775 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:32:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:32:58.610+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:32:58.775 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:32:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:32:58.610+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:32:58.775 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:32:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:32:58.440+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:32:58.776 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:32:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:32:58.440+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:32:58.776 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:32:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:32:58.440+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:32:58.776 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:32:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:32:58.440+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:32:59.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:32:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:32:58.773+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:32:59.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:32:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:32:58.773+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:32:59.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:32:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:32:58.773+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:32:59.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:32:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:32:58.774+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:32:59.771 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:32:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:32:59.451+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:32:59.771 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:32:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:32:59.451+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:32:59.771 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:32:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:32:59.451+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:32:59.771 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:32:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:32:59.451+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:32:59.772 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:32:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:32:59.596+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:32:59.772 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:32:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:32:59.596+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:32:59.772 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:32:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:32:59.596+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:32:59.772 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:32:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:32:59.596+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:33:00.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:32:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:32:59.769+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:33:00.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:32:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:32:59.769+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:33:00.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:32:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:32:59.769+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:33:00.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:32:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:32:59.769+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:33:00.497 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:33:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:33:00.244+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 9875 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:33:00.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:33:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:33:00.496+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:33:00.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:33:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:33:00.496+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:33:00.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:33:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:33:00.496+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:33:00.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:33:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:33:00.497+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:33:00.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:33:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:33:00.720+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:33:00.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:33:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:33:00.720+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:33:00.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:33:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:33:00.720+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:33:00.821 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:33:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:33:00.721+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:33:00.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:33:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:33:00.587+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:33:00.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:33:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:33:00.587+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:33:00.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:33:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:33:00.588+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:33:00.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:33:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:33:00.588+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:33:01.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:33:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:33:01.724+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:33:01.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:33:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:33:01.724+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:33:01.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:33:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:33:01.724+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:33:01.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:33:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:33:01.724+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:33:01.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:33:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:33:01.525+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:33:01.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:33:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:33:01.525+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:33:01.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:33:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:33:01.525+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:33:01.821 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:33:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:33:01.526+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:33:01.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:33:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:33:01.539+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:33:01.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:33:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:33:01.539+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:33:01.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:33:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:33:01.539+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:33:01.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:33:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:33:01.540+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:33:02.767 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:33:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:33:02.563+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:33:02.768 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:33:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:33:02.563+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:33:02.768 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:33:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:33:02.563+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:33:02.768 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:33:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:33:02.563+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:33:02.768 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:33:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:33:02.514+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:33:02.768 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:33:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:33:02.514+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:33:02.768 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:33:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:33:02.514+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:33:02.768 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:33:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:33:02.514+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:33:03.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:33:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:33:02.766+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:33:03.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:33:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:33:02.766+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:33:03.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:33:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:33:02.766+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:33:03.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:33:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:33:02.766+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:33:03.795 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:33:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:33:03.507+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:33:03.795 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:33:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:33:03.507+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:33:03.796 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:33:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:33:03.507+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:33:03.796 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:33:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:33:03.507+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:33:03.796 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:33:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:33:03.578+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:33:03.796 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:33:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:33:03.578+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:33:03.796 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:33:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:33:03.578+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:33:03.796 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:33:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:33:03.578+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:33:04.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:33:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:33:03.794+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:33:04.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:33:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:33:03.794+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:33:04.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:33:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:33:03.794+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:33:04.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:33:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:33:03.795+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:33:04.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:33:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:33:04.532+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:33:04.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:33:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:33:04.532+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:33:04.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:33:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:33:04.532+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:33:04.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:33:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:33:04.533+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:33:04.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:33:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:33:04.561+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:33:04.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:33:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:33:04.561+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:33:04.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:33:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:33:04.561+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:33:04.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:33:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:33:04.561+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:33:05.247 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:33:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:33:04.827+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:33:05.248 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:33:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:33:04.827+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:33:05.248 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:33:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:33:04.827+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:33:05.248 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:33:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:33:04.828+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:33:05.518 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:33:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:33:05.247+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 9970 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:33:05.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:33:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:33:05.611+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:33:05.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:33:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:33:05.611+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:33:05.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:33:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:33:05.611+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:33:05.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:33:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:33:05.612+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:33:05.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:33:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:33:05.517+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:33:05.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:33:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:33:05.517+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:33:05.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:33:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:33:05.517+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:33:05.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:33:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:33:05.518+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:33:06.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:33:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:33:05.849+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:33:06.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:33:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:33:05.849+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:33:06.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:33:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:33:05.849+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:33:06.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:33:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:33:05.850+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:33:06.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:33:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:33:06.587+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:33:06.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:33:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:33:06.587+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:33:06.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:33:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:33:06.587+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:33:06.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:33:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:33:06.587+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:33:06.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:33:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:33:06.482+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:33:06.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:33:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:33:06.482+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:33:06.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:33:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:33:06.482+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:33:06.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:33:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:33:06.482+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:33:07.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:33:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:33:06.869+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:33:07.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:33:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:33:06.869+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:33:07.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:33:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:33:06.869+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:33:07.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:33:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:33:06.869+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:33:07.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:33:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:33:07.585+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:33:07.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:33:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:33:07.585+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:33:07.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:33:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:33:07.585+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:33:07.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:33:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:33:07.586+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:33:07.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:33:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:33:07.504+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:33:07.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:33:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:33:07.504+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:33:07.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:33:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:33:07.504+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:33:07.821 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:33:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:33:07.505+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:33:08.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:33:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:33:07.833+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:33:08.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:33:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:33:07.833+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:33:08.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:33:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:33:07.833+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:33:08.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:33:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:33:07.834+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:33:08.799 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:33:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:33:08.536+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:33:08.799 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:33:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:33:08.536+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:33:08.799 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:33:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:33:08.536+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:33:08.800 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:33:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:33:08.537+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:33:08.800 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:33:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:33:08.568+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:33:08.800 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:33:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:33:08.568+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:33:08.800 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:33:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:33:08.568+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:33:08.800 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:33:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:33:08.568+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:33:09.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:33:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:33:08.798+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:33:09.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:33:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:33:08.798+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:33:09.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:33:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:33:08.798+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:33:09.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:33:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:33:08.799+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:33:09.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:33:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:33:09.556+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:33:09.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:33:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:33:09.556+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:33:09.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:33:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:33:09.556+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:33:09.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:33:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:33:09.556+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:33:09.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:33:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:33:09.789+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:33:09.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:33:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:33:09.789+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:33:09.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:33:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:33:09.789+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:33:09.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:33:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:33:09.790+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:33:09.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:33:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:33:09.573+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:33:09.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:33:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:33:09.573+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:33:09.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:33:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:33:09.573+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:33:09.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:33:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:33:09.573+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:33:10.523 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:33:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:33:10.248+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 10065 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:33:10.811 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:33:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:33:10.560+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:33:10.812 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:33:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:33:10.560+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:33:10.812 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:33:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:33:10.560+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:33:10.812 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:33:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:33:10.560+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:33:10.812 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:33:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:33:10.522+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:33:10.812 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:33:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:33:10.522+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:33:10.813 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:33:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:33:10.522+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:33:10.813 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:33:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:33:10.523+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:33:11.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:33:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:33:10.822+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:33:11.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:33:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:33:10.822+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:33:11.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:33:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:33:10.822+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:33:11.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:33:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:33:10.823+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:33:11.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:33:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:33:11.553+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:33:11.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:33:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:33:11.553+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:33:11.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:33:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:33:11.553+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:33:11.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:33:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:33:11.554+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:33:11.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:33:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:33:11.481+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:33:11.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:33:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:33:11.481+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:33:11.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:33:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:33:11.481+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:33:11.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:33:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:33:11.482+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:33:12.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:33:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:33:11.863+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:33:12.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:33:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:33:11.863+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:33:12.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:33:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:33:11.863+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:33:12.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:33:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:33:11.864+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:33:12.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:33:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:33:12.558+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:33:12.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:33:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:33:12.558+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:33:12.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:33:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:33:12.558+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:33:12.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:33:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:33:12.558+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:33:12.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:33:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:33:12.527+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:33:12.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:33:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:33:12.527+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:33:12.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:33:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:33:12.527+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:33:12.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:33:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:33:12.529+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:33:13.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:33:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:33:12.849+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:33:13.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:33:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:33:12.849+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:33:13.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:33:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:33:12.849+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:33:13.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:33:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:33:12.850+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:33:13.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:33:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:33:13.592+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:33:13.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:33:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:33:13.592+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:33:13.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:33:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:33:13.592+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:33:13.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:33:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:33:13.593+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:33:13.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:33:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:33:13.564+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:33:13.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:33:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:33:13.564+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:33:13.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:33:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:33:13.564+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:33:13.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:33:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:33:13.565+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:33:14.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:33:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:33:13.820+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:33:14.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:33:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:33:13.820+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:33:14.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:33:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:33:13.821+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:33:14.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:33:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:33:13.822+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:33:14.811 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:33:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:33:14.545+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:33:14.811 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:33:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:33:14.545+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:33:14.812 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:33:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:33:14.545+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:33:14.812 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:33:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:33:14.545+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:33:14.812 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:33:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:33:14.547+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:33:14.812 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:33:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:33:14.547+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:33:14.812 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:33:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:33:14.547+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:33:14.812 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:33:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:33:14.548+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:33:15.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:33:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:33:14.811+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:33:15.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:33:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:33:14.811+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:33:15.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:33:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:33:14.811+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:33:15.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:33:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:33:14.812+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:33:15.517 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:33:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:33:15.249+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 10159 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:33:15.789 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:33:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:33:15.516+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:33:15.789 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:33:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:33:15.516+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:33:15.789 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:33:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:33:15.516+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:33:15.789 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:33:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:33:15.517+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:33:15.790 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:33:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:33:15.581+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:33:15.790 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:33:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:33:15.581+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:33:15.790 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:33:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:33:15.581+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:33:15.790 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:33:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:33:15.581+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:33:16.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:33:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:33:15.788+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:33:16.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:33:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:33:15.788+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:33:16.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:33:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:33:15.788+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:33:16.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:33:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:33:15.789+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:33:16.803 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:33:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:33:16.594+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:33:16.803 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:33:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:33:16.594+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:33:16.803 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:33:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:33:16.594+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:33:16.804 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:33:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:33:16.595+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:33:16.804 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:33:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:33:16.488+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:33:16.804 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:33:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:33:16.488+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:33:16.804 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:33:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:33:16.488+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:33:16.804 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:33:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:33:16.489+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:33:17.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:33:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:33:16.802+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:33:17.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:33:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:33:16.802+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:33:17.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:33:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:33:16.802+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:33:17.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:33:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:33:16.803+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:33:17.760 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:33:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:33:17.620+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:33:17.761 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:33:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:33:17.620+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:33:17.761 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:33:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:33:17.620+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:33:17.761 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:33:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:33:17.620+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:33:17.761 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:33:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:33:17.508+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:33:17.761 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:33:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:33:17.508+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:33:17.762 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:33:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:33:17.508+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:33:17.762 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:33:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:33:17.508+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:33:18.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:33:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:33:17.760+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:33:18.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:33:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:33:17.760+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:33:18.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:33:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:33:17.760+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:33:18.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:33:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:33:17.761+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:33:18.782 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:33:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:33:18.478+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:33:18.782 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:33:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:33:18.478+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:33:18.782 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:33:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:33:18.478+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:33:18.782 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:33:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:33:18.479+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:33:18.782 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:33:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:33:18.579+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:33:18.783 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:33:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:33:18.579+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:33:18.783 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:33:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:33:18.579+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:33:18.783 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:33:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:33:18.580+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:33:19.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:33:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:33:18.781+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:33:19.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:33:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:33:18.781+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:33:19.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:33:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:33:18.781+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:33:19.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:33:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:33:18.782+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:33:19.744 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:33:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:33:19.483+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:33:19.745 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:33:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:33:19.483+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:33:19.745 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:33:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:33:19.483+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:33:19.745 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:33:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:33:19.484+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:33:19.745 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:33:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:33:19.584+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:33:19.745 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:33:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:33:19.584+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:33:19.745 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:33:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:33:19.584+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:33:19.745 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:33:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:33:19.584+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:33:20.027 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:33:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:33:19.743+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:33:20.028 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:33:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:33:19.744+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:33:20.028 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:33:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:33:19.744+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:33:20.028 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:33:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:33:19.745+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:33:20.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:33:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:33:20.250+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 10252 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:33:20.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:33:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:33:20.467+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:33:20.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:33:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:33:20.467+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:33:20.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:33:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:33:20.467+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:33:20.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:33:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:33:20.468+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:33:21.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:33:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:33:20.720+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:33:21.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:33:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:33:20.720+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:33:21.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:33:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:33:20.720+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:33:21.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:33:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:33:20.721+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:33:21.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:33:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:33:20.634+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:33:21.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:33:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:33:20.634+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:33:21.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:33:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:33:20.634+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:33:21.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:33:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:33:20.634+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:33:21.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:33:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:33:21.740+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:33:21.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:33:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:33:21.740+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:33:21.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:33:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:33:21.740+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:33:21.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:33:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:33:21.741+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:33:21.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:33:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:33:21.671+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:33:21.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:33:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:33:21.671+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:33:21.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:33:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:33:21.671+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:33:21.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:33:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:33:21.671+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:33:21.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:33:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:33:21.503+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:33:21.821 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:33:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:33:21.503+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:33:21.821 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:33:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:33:21.503+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:33:21.821 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:33:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:33:21.504+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:33:22.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:33:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:33:22.694+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:33:22.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:33:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:33:22.694+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:33:22.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:33:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:33:22.694+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:33:22.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:33:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:33:22.695+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:33:22.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:33:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:33:22.499+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:33:22.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:33:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:33:22.499+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:33:22.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:33:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:33:22.499+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:33:22.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:33:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:33:22.500+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:33:22.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:33:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:33:22.692+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:33:22.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:33:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:33:22.692+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:33:22.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:33:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:33:22.692+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:33:22.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:33:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:33:22.692+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:33:23.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:33:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:33:23.720+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:33:23.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:33:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:33:23.720+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:33:23.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:33:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:33:23.720+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:33:23.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:33:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:33:23.720+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:33:23.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:33:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:33:23.646+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:33:23.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:33:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:33:23.646+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:33:23.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:33:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:33:23.646+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:33:23.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:33:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:33:23.647+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:33:23.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:33:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:33:23.515+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:33:23.821 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:33:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:33:23.515+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:33:23.821 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:33:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:33:23.515+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:33:23.821 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:33:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:33:23.516+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:33:24.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:33:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:33:24.685+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:33:24.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:33:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:33:24.685+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:33:24.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:33:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:33:24.685+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:33:24.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:33:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:33:24.686+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:33:24.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:33:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:33:24.511+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:33:24.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:33:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:33:24.511+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:33:24.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:33:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:33:24.511+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:33:24.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:33:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:33:24.511+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:33:24.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:33:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:33:24.701+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:33:24.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:33:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:33:24.701+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:33:24.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:33:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:33:24.701+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:33:24.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:33:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:33:24.701+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:33:25.547 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:33:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:33:25.251+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 10346 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:33:25.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:33:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:33:25.546+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:33:25.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:33:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:33:25.546+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:33:25.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:33:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:33:25.546+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:33:25.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:33:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:33:25.547+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:33:25.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:33:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:33:25.746+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:33:25.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:33:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:33:25.746+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:33:25.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:33:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:33:25.746+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:33:25.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:33:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:33:25.746+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:33:25.821 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:33:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:33:25.636+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:33:25.821 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:33:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:33:25.636+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:33:25.821 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:33:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:33:25.636+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:33:25.821 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:33:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:33:25.638+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:33:27.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:33:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:33:26.575+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:33:27.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:33:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:33:26.575+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:33:27.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:33:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:33:26.575+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:33:27.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:33:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:33:26.576+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:33:27.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:33:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:33:26.733+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:33:27.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:33:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:33:26.733+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:33:27.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:33:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:33:26.733+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:33:27.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:33:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:33:26.733+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:33:27.071 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:33:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:33:26.591+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:33:27.071 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:33:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:33:26.591+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:33:27.071 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:33:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:33:26.591+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:33:27.071 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:33:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:33:26.592+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:33:27.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:33:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:33:27.769+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:33:27.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:33:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:33:27.769+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:33:27.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:33:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:33:27.769+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:33:27.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:33:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:33:27.769+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:33:27.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:33:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:33:27.540+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:33:27.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:33:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:33:27.540+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:33:27.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:33:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:33:27.540+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:33:27.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:33:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:33:27.541+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:33:27.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:33:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:33:27.615+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:33:27.821 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:33:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:33:27.615+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:33:27.821 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:33:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:33:27.615+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:33:27.821 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:33:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:33:27.616+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:33:28.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:33:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:33:28.797+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:33:28.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:33:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:33:28.797+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:33:28.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:33:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:33:28.797+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:33:28.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:33:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:33:28.798+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:33:28.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:33:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:33:28.558+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:33:28.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:33:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:33:28.558+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:33:28.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:33:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:33:28.558+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:33:28.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:33:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:33:28.559+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:33:28.821 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:33:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:33:28.657+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:33:28.821 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:33:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:33:28.657+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:33:28.821 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:33:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:33:28.657+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:33:28.821 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:33:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:33:28.658+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:33:29.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:33:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:33:29.764+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:33:29.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:33:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:33:29.764+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:33:29.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:33:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:33:29.764+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:33:29.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:33:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:33:29.765+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:33:29.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:33:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:33:29.542+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:33:29.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:33:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:33:29.542+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:33:29.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:33:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:33:29.542+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:33:29.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:33:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:33:29.543+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:33:29.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:33:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:33:29.646+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:33:29.821 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:33:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:33:29.646+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:33:29.821 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:33:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:33:29.646+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:33:29.821 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:33:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:33:29.646+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:33:30.556 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:33:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:33:30.253+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 10440 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:33:30.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:33:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:33:30.754+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:33:30.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:33:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:33:30.754+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:33:30.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:33:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:33:30.754+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:33:30.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:33:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:33:30.754+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:33:30.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:33:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:33:30.555+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:33:30.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:33:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:33:30.555+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:33:30.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:33:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:33:30.555+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:33:30.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:33:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:33:30.556+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:33:30.821 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:33:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:33:30.664+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:33:30.821 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:33:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:33:30.664+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:33:30.821 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:33:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:33:30.664+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:33:30.821 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:33:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:33:30.665+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:33:32.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:33:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:33:31.573+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:33:32.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:33:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:33:31.573+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:33:32.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:33:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:33:31.573+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:33:32.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:33:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:33:31.573+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:33:32.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:33:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:33:31.784+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:33:32.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:33:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:33:31.784+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:33:32.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:33:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:33:31.784+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:33:32.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:33:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:33:31.785+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:33:32.071 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:33:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:33:31.649+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:33:32.071 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:33:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:33:31.649+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:33:32.071 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:33:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:33:31.649+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:33:32.071 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:33:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:33:31.650+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:33:33.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:33:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:33:32.796+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:33:33.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:33:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:33:32.796+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:33:33.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:33:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:33:32.796+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:33:33.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:33:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:33:32.796+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:33:33.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:33:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:33:32.590+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:33:33.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:33:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:33:32.590+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:33:33.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:33:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:33:32.590+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:33:33.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:33:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:33:32.591+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:33:33.071 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:33:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:33:32.612+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:33:33.071 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:33:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:33:32.612+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:33:33.071 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:33:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:33:32.612+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:33:33.071 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:33:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:33:32.614+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:33:33.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:33:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:33:33.804+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:33:33.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:33:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:33:33.804+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:33:33.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:33:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:33:33.804+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:33:33.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:33:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:33:33.805+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:33:33.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:33:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:33:33.602+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:33:33.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:33:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:33:33.602+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:33:33.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:33:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:33:33.602+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:33:33.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:33:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:33:33.603+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:33:33.821 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:33:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:33:33.614+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:33:33.821 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:33:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:33:33.614+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:33:33.821 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:33:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:33:33.614+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:33:33.821 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:33:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:33:33.615+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:33:35.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:33:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:33:34.815+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:33:35.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:33:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:33:34.815+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:33:35.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:33:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:33:34.815+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:33:35.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:33:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:33:34.815+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:33:35.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:33:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:33:34.636+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:33:35.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:33:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:33:34.636+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:33:35.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:33:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:33:34.636+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:33:35.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:33:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:33:34.637+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:33:35.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:33:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:33:34.598+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:33:35.071 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:33:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:33:34.598+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:33:35.071 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:33:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:33:34.598+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:33:35.071 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:33:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:33:34.599+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:33:35.557 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:33:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:33:35.254+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 10532 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:33:35.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:33:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:33:35.556+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:33:35.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:33:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:33:35.556+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:33:35.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:33:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:33:35.556+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:33:35.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:33:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:33:35.557+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:33:35.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:33:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:33:35.629+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:33:35.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:33:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:33:35.629+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:33:35.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:33:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:33:35.629+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:33:35.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:33:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:33:35.630+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:33:36.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:33:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:33:35.847+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:33:36.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:33:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:33:35.847+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:33:36.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:33:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:33:35.847+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:33:36.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:33:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:33:35.848+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:33:36.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:33:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:33:36.594+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:33:36.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:33:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:33:36.594+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:33:36.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:33:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:33:36.594+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:33:36.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:33:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:33:36.594+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:33:36.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:33:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:33:36.537+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:33:36.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:33:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:33:36.537+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:33:36.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:33:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:33:36.537+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:33:36.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:33:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:33:36.539+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:33:37.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:33:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:33:36.885+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:33:37.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:33:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:33:36.885+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:33:37.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:33:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:33:36.885+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:33:37.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:33:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:33:36.885+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:33:37.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:33:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:33:37.628+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:33:37.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:33:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:33:37.628+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:33:37.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:33:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:33:37.628+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:33:37.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:33:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:33:37.629+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:33:37.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:33:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:33:37.491+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:33:37.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:33:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:33:37.491+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:33:37.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:33:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:33:37.491+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:33:37.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:33:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:33:37.492+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:33:38.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:33:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:33:37.879+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:33:38.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:33:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:33:37.879+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:33:38.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:33:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:33:37.879+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:33:38.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:33:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:33:37.879+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:33:38.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:33:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:33:38.635+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:33:38.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:33:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:33:38.635+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:33:38.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:33:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:33:38.635+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:33:38.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:33:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:33:38.636+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:33:38.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:33:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:33:38.477+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:33:38.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:33:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:33:38.477+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:33:38.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:33:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:33:38.477+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:33:38.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:33:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:33:38.478+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:33:39.116 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:33:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:33:38.849+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:33:39.116 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:33:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:33:38.849+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:33:39.116 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:33:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:33:38.849+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:33:39.116 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:33:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:33:38.849+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:33:39.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:33:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:33:39.504+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:33:39.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:33:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:33:39.504+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:33:39.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:33:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:33:39.504+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:33:39.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:33:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:33:39.504+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:33:39.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:33:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:33:39.681+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:33:39.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:33:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:33:39.681+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:33:39.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:33:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:33:39.681+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:33:39.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:33:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:33:39.681+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:33:40.256 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:33:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:33:39.823+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:33:40.256 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:33:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:33:39.823+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:33:40.256 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:33:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:33:39.823+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:33:40.256 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:33:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:33:39.823+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:33:40.548 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:33:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:33:40.255+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 10625 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:33:40.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:33:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:33:40.548+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:33:40.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:33:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:33:40.548+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:33:40.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:33:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:33:40.548+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:33:40.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:33:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:33:40.549+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:33:40.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:33:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:33:40.644+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:33:40.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:33:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:33:40.644+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:33:40.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:33:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:33:40.644+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:33:40.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:33:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:33:40.645+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:33:40.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:33:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:33:40.782+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:33:40.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:33:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:33:40.782+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:33:40.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:33:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:33:40.782+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:33:40.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:33:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:33:40.782+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:33:41.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:33:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:33:41.679+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:33:41.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:33:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:33:41.679+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:33:41.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:33:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:33:41.679+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:33:41.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:33:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:33:41.680+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:33:41.821 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:33:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:33:41.584+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:33:41.821 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:33:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:33:41.584+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:33:41.821 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:33:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:33:41.584+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:33:41.821 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:33:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:33:41.584+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:33:42.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:33:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:33:41.832+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:33:42.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:33:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:33:41.832+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:33:42.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:33:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:33:41.832+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:33:42.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:33:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:33:41.832+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:33:42.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:33:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:33:42.552+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:33:42.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:33:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:33:42.552+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:33:42.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:33:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:33:42.552+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:33:42.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:33:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:33:42.552+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:33:42.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:33:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:33:42.729+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:33:42.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:33:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:33:42.729+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:33:42.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:33:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:33:42.729+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:33:42.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:33:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:33:42.731+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:33:43.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:33:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:33:42.841+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:33:43.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:33:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:33:42.841+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:33:43.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:33:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:33:42.841+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:33:43.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:33:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:33:42.841+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:33:43.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:33:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:33:43.533+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:33:43.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:33:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:33:43.533+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:33:43.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:33:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:33:43.533+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:33:43.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:33:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:33:43.534+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:33:43.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:33:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:33:43.694+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:33:43.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:33:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:33:43.694+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:33:43.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:33:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:33:43.694+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:33:43.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:33:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:33:43.695+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:33:44.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:33:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:33:43.837+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:33:44.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:33:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:33:43.837+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:33:44.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:33:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:33:43.837+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:33:44.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:33:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:33:43.837+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:33:44.603 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:33:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:33:44.559+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:33:44.603 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:33:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:33:44.559+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:33:44.603 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:33:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:33:44.559+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:33:44.604 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:33:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:33:44.560+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:33:44.868 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:33:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:33:44.723+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:33:44.868 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:33:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:33:44.723+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:33:44.868 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:33:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:33:44.723+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:33:44.868 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:33:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:33:44.724+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:33:45.257 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:33:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:33:44.867+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:33:45.258 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:33:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:33:44.867+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:33:45.258 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:33:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:33:44.867+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:33:45.258 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:33:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:33:44.868+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:33:45.541 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:33:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:33:45.256+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 10718 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:33:45.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:33:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:33:45.541+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:33:45.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:33:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:33:45.541+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:33:45.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:33:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:33:45.541+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:33:45.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:33:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:33:45.542+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:33:45.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:33:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:33:45.700+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:33:45.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:33:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:33:45.700+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:33:45.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:33:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:33:45.700+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:33:45.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:33:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:33:45.701+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:33:46.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:33:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:33:45.888+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:33:46.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:33:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:33:45.888+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:33:46.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:33:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:33:45.888+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:33:46.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:33:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:33:45.888+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:33:46.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:33:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:33:46.654+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:33:46.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:33:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:33:46.654+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:33:46.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:33:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:33:46.654+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:33:46.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:33:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:33:46.655+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:33:46.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:33:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:33:46.553+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:33:46.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:33:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:33:46.553+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:33:46.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:33:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:33:46.553+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:33:46.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:33:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:33:46.554+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:33:47.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:33:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:33:46.896+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:33:47.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:33:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:33:46.896+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:33:47.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:33:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:33:46.896+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:33:47.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:33:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:33:46.897+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:33:47.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:33:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:33:47.704+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:33:47.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:33:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:33:47.704+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:33:47.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:33:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:33:47.704+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:33:47.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:33:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:33:47.705+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:33:47.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:33:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:33:47.531+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:33:47.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:33:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:33:47.531+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:33:47.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:33:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:33:47.531+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:33:47.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:33:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:33:47.532+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:33:48.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:33:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:33:47.887+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:33:48.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:33:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:33:47.887+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:33:48.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:33:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:33:47.887+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:33:48.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:33:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:33:47.887+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:33:48.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:33:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:33:48.489+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:33:48.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:33:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:33:48.489+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:33:48.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:33:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:33:48.489+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:33:48.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:33:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:33:48.489+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:33:48.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:33:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:33:48.701+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:33:48.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:33:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:33:48.701+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:33:48.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:33:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:33:48.701+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:33:48.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:33:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:33:48.701+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:33:49.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:33:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:33:48.906+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:33:49.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:33:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:33:48.906+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:33:49.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:33:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:33:48.906+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:33:49.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:33:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:33:48.906+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:33:49.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:33:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:33:49.679+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:33:49.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:33:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:33:49.679+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:33:49.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:33:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:33:49.679+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:33:49.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:33:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:33:49.680+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:33:49.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:33:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:33:49.497+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:33:49.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:33:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:33:49.497+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:33:49.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:33:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:33:49.497+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:33:49.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:33:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:33:49.498+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:33:50.258 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:33:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:33:49.938+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:33:50.259 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:33:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:33:49.938+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:33:50.259 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:33:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:33:49.939+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:33:50.259 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:33:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:33:49.939+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:33:50.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:33:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:33:50.258+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 10813 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:33:50.569 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:33:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:33:50.477+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:33:50.569 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:33:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:33:50.477+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:33:50.570 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:33:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:33:50.477+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:33:50.570 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:33:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:33:50.478+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:33:50.969 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:33:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:33:50.630+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:33:50.969 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:33:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:33:50.630+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:33:50.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:33:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:33:50.630+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:33:50.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:33:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:33:50.631+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:33:51.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:33:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:33:50.968+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:33:51.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:33:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:33:50.968+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:33:51.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:33:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:33:50.968+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:33:51.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:33:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:33:50.969+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:33:51.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:33:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:33:51.520+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:33:51.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:33:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:33:51.520+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:33:51.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:33:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:33:51.520+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:33:51.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:33:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:33:51.521+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:33:51.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:33:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:33:51.650+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:33:51.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:33:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:33:51.650+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:33:51.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:33:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:33:51.650+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:33:51.821 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:33:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:33:51.651+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:33:52.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:33:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:33:51.972+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:33:52.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:33:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:33:51.972+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:33:52.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:33:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:33:51.972+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:33:52.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:33:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:33:51.972+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:33:52.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:33:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:33:52.539+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:33:52.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:33:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:33:52.539+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:33:52.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:33:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:33:52.539+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:33:52.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:33:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:33:52.539+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:33:52.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:33:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:33:52.652+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:33:52.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:33:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:33:52.652+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:33:52.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:33:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:33:52.652+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:33:52.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:33:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:33:52.653+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:33:53.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:33:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:33:53.013+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:33:53.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:33:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:33:53.013+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:33:53.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:33:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:33:53.013+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:33:53.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:33:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:33:53.013+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:33:53.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:33:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:33:53.702+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:33:53.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:33:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:33:53.702+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:33:53.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:33:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:33:53.702+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:33:53.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:33:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:33:53.703+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:33:53.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:33:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:33:53.573+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:33:53.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:33:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:33:53.573+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:33:53.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:33:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:33:53.573+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:33:53.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:33:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:33:53.574+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:33:54.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:33:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:33:53.985+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:33:54.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:33:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:33:53.985+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:33:54.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:33:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:33:53.985+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:33:54.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:33:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:33:53.985+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:33:54.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:33:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:33:54.720+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:33:54.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:33:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:33:54.720+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:33:54.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:33:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:33:54.720+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:33:54.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:33:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:33:54.721+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:33:54.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:33:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:33:54.565+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:33:54.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:33:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:33:54.565+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:33:54.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:33:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:33:54.565+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:33:54.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:33:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:33:54.566+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:33:55.271 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:33:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:33:55.258+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 10907 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:33:55.272 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:33:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:33:55.018+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:33:55.272 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:33:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:33:55.018+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:33:55.272 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:33:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:33:55.018+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:33:55.272 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:33:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:33:55.018+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:33:55.971 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:33:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:33:55.609+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:33:55.971 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:33:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:33:55.609+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:33:55.971 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:33:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:33:55.609+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:33:55.971 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:33:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:33:55.610+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:33:55.972 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:33:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:33:55.702+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:33:55.972 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:33:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:33:55.702+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:33:55.972 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:33:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:33:55.702+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:33:55.972 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:33:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:33:55.703+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:33:56.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:33:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:33:55.969+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:33:56.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:33:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:33:55.969+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:33:56.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:33:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:33:55.969+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:33:56.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:33:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:33:55.969+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:33:57.014 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:33:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:33:56.640+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:33:57.015 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:33:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:33:56.640+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:33:57.015 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:33:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:33:56.640+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:33:57.015 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:33:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:33:56.641+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:33:57.015 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:33:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:33:56.683+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:33:57.015 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:33:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:33:56.683+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:33:57.016 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:33:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:33:56.683+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:33:57.016 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:33:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:33:56.683+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:33:57.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:33:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:33:57.013+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:33:57.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:33:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:33:57.013+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:33:57.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:33:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:33:57.013+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:33:57.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:33:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:33:57.013+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:33:58.021 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:33:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:33:57.647+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:33:58.022 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:33:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:33:57.647+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:33:58.022 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:33:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:33:57.647+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:33:58.022 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:33:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:33:57.648+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:33:58.022 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:33:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:33:57.678+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:33:58.022 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:33:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:33:57.678+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:33:58.023 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:33:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:33:57.678+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:33:58.023 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:33:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:33:57.678+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:33:58.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:33:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:33:58.019+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:33:58.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:33:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:33:58.019+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:33:58.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:33:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:33:58.019+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:33:58.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:33:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:33:58.020+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:33:59.008 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:33:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:33:58.698+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:33:59.008 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:33:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:33:58.698+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:33:59.008 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:33:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:33:58.698+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:33:59.008 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:33:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:33:58.699+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:33:59.008 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:33:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:33:58.655+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:33:59.009 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:33:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:33:58.655+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:33:59.009 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:33:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:33:58.655+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:33:59.009 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:33:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:33:58.656+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:33:59.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:33:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:33:59.006+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:33:59.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:33:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:33:59.006+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:33:59.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:33:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:33:59.006+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:33:59.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:33:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:33:59.006+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:33:59.966 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:33:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:33:59.661+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:33:59.966 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:33:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:33:59.661+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:33:59.966 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:33:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:33:59.661+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:33:59.966 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:33:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:33:59.661+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:33:59.966 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:33:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:33:59.699+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:33:59.967 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:33:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:33:59.699+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:33:59.967 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:33:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:33:59.699+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:33:59.967 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:33:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:33:59.700+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:34:00.261 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:33:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:33:59.964+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:34:00.261 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:33:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:33:59.964+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:34:00.261 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:33:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:33:59.964+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:34:00.261 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:33:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:33:59.964+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:34:00.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:34:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:34:00.259+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 11001 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:34:00.967 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:34:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:34:00.644+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:34:00.967 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:34:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:34:00.644+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:34:00.968 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:34:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:34:00.644+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:34:00.968 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:34:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:34:00.644+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:34:00.968 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:34:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:34:00.731+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:34:00.968 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:34:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:34:00.731+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:34:00.968 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:34:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:34:00.731+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:34:00.969 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:34:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:34:00.732+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:34:01.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:34:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:34:00.966+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:34:01.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:34:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:34:00.966+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:34:01.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:34:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:34:00.966+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:34:01.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:34:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:34:00.966+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:34:01.954 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:34:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:34:01.770+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:34:01.954 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:34:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:34:01.770+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:34:01.955 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:34:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:34:01.770+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:34:01.955 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:34:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:34:01.771+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:34:01.955 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:34:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:34:01.611+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:34:01.955 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:34:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:34:01.611+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:34:01.955 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:34:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:34:01.611+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:34:01.955 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:34:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:34:01.611+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:34:02.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:34:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:34:01.953+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:34:02.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:34:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:34:01.953+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:34:02.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:34:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:34:01.953+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:34:02.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:34:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:34:01.954+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:34:02.978 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:34:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:34:02.586+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:34:02.978 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:34:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:34:02.586+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:34:02.979 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:34:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:34:02.586+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:34:02.979 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:34:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:34:02.587+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:34:02.979 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:34:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:34:02.727+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:34:02.980 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:34:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:34:02.727+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:34:02.980 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:34:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:34:02.727+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:34:02.980 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:34:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:34:02.728+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:34:03.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:34:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:34:02.977+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:34:03.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:34:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:34:02.977+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:34:03.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:34:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:34:02.977+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:34:03.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:34:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:34:02.977+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:34:03.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:34:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:34:03.705+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:34:03.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:34:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:34:03.705+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:34:03.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:34:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:34:03.705+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:34:03.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:34:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:34:03.706+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:34:03.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:34:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:34:03.633+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:34:03.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:34:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:34:03.633+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:34:03.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:34:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:34:03.633+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:34:03.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:34:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:34:03.634+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:34:04.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:34:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:34:03.955+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:34:04.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:34:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:34:03.955+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:34:04.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:34:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:34:03.955+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:34:04.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:34:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:34:03.956+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:34:04.985 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:34:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:34:04.594+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:34:04.986 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:34:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:34:04.594+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:34:04.986 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:34:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:34:04.594+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:34:04.986 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:34:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:34:04.595+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:34:04.987 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:34:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:34:04.657+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:34:04.987 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:34:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:34:04.657+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:34:04.987 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:34:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:34:04.657+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:34:04.987 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:34:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:34:04.658+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:34:05.263 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:34:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:34:04.984+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:34:05.263 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:34:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:34:04.984+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:34:05.263 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:34:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:34:04.984+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:34:05.263 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:34:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:34:04.985+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:34:05.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:34:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:34:05.261+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 11095 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:34:05.949 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:34:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:34:05.683+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:34:05.949 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:34:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:34:05.684+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:34:05.949 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:34:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:34:05.684+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:34:05.949 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:34:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:34:05.684+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:34:05.950 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:34:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:34:05.573+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:34:05.950 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:34:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:34:05.573+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:34:05.950 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:34:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:34:05.573+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:34:05.950 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:34:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:34:05.574+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:34:06.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:34:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:34:05.948+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:34:06.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:34:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:34:05.948+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:34:06.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:34:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:34:05.948+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:34:06.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:34:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:34:05.948+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:34:06.926 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:34:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:34:06.669+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:34:06.927 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:34:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:34:06.669+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:34:06.927 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:34:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:34:06.669+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:34:06.927 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:34:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:34:06.670+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:34:06.927 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:34:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:34:06.588+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:34:06.927 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:34:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:34:06.588+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:34:06.928 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:34:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:34:06.588+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:34:06.928 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:34:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:34:06.588+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:34:07.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:34:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:34:06.926+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:34:07.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:34:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:34:06.926+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:34:07.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:34:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:34:06.926+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:34:07.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:34:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:34:06.926+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:34:07.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:34:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:34:07.697+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:34:07.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:34:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:34:07.697+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:34:07.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:34:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:34:07.697+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:34:07.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:34:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:34:07.698+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:34:07.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:34:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:34:07.561+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:34:07.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:34:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:34:07.561+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:34:07.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:34:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:34:07.561+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:34:07.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:34:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:34:07.561+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:34:08.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:34:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:34:07.963+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:34:08.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:34:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:34:07.963+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:34:08.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:34:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:34:07.963+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:34:08.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:34:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:34:07.964+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:34:08.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:34:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:34:08.549+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:34:08.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:34:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:34:08.549+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:34:08.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:34:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:34:08.549+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:34:08.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:34:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:34:08.550+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:34:08.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:34:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:34:08.707+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:34:08.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:34:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:34:08.707+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:34:08.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:34:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:34:08.707+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:34:08.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:34:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:34:08.708+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:34:09.179 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:34:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:34:08.921+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:34:09.179 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:34:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:34:08.921+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:34:09.180 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:34:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:34:08.921+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:34:09.180 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:34:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:34:08.922+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:34:09.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:34:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:34:09.537+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:34:09.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:34:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:34:09.537+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:34:09.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:34:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:34:09.537+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:34:09.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:34:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:34:09.537+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:34:09.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:34:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:34:09.660+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:34:09.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:34:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:34:09.660+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:34:09.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:34:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:34:09.660+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:34:09.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:34:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:34:09.660+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:34:10.263 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:34:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:34:09.946+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:34:10.264 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:34:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:34:09.946+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:34:10.264 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:34:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:34:09.946+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:34:10.264 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:34:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:34:09.947+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:34:10.542 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:34:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:34:10.263+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 11190 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:34:10.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:34:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:34:10.675+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:34:10.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:34:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:34:10.675+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:34:10.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:34:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:34:10.675+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:34:10.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:34:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:34:10.676+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:34:10.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:34:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:34:10.542+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:34:10.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:34:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:34:10.542+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:34:10.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:34:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:34:10.542+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:34:10.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:34:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:34:10.542+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:34:11.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:34:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:34:10.986+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:34:11.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:34:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:34:10.986+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:34:11.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:34:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:34:10.986+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:34:11.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:34:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:34:10.986+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:34:11.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:34:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:34:11.654+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:34:11.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:34:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:34:11.654+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:34:11.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:34:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:34:11.654+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:34:11.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:34:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:34:11.655+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:34:11.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:34:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:34:11.517+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:34:11.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:34:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:34:11.517+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:34:11.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:34:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:34:11.517+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:34:11.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:34:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:34:11.518+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:34:12.250 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:34:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:34:11.945+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:34:12.251 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:34:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:34:11.945+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:34:12.251 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:34:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:34:11.945+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:34:12.251 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:34:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:34:11.946+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:34:12.731 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-18T01:34:12.732 DEBUG:tasks.cephfs.mount:Force/lazy unmounting on client.0 2024-03-18T01:34:12.732 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T01:34:12.732 DEBUG:teuthology.orchestra.run.smithi029:> sudo umount --lazy --force /home/ubuntu/cephtest/mnt.0 2024-03-18T01:34:12.813 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_attr_setschedparam/speculative/3-1.c: Cannot open: Input/output error 2024-03-18T01:34:12.814 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_attr_setschedparam/speculative/3-2.c: Cannot open: Input/output error 2024-03-18T01:34:12.814 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_attr_setschedparam/speculative: Cannot utime: Input/output error 2024-03-18T01:34:12.814 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_attr_setschedparam: Cannot utime: Input/output error 2024-03-18T01:34:12.814 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigtimedwait: Cannot mkdir: Input/output error 2024-03-18T01:34:12.815 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigtimedwait/coverage.txt: Cannot open: Input/output error 2024-03-18T01:34:12.815 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigtimedwait/2-1.c: Cannot open: Input/output error 2024-03-18T01:34:12.815 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigtimedwait/5-1.c: Cannot open: Input/output error 2024-03-18T01:34:12.815 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigtimedwait/6-1.c: Cannot open: Input/output error 2024-03-18T01:34:12.815 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigtimedwait/1-1.c: Cannot open: Input/output error 2024-03-18T01:34:12.815 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigtimedwait/assertions.xml: Cannot open: Input/output error 2024-03-18T01:34:12.815 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigtimedwait/4-1.c: Cannot open: Input/output error 2024-03-18T01:34:12.815 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sem_post: Cannot mkdir: Input/output error 2024-03-18T01:34:12.816 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sem_post/coverage.txt: Cannot open: Input/output error 2024-03-18T01:34:12.816 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sem_post/1-2.c: Cannot open: Input/output error 2024-03-18T01:34:12.816 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sem_post/2-1.c: Cannot open: Input/output error 2024-03-18T01:34:12.816 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sem_post/5-1.c: Cannot open: Input/output error 2024-03-18T01:34:12.816 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sem_post/6-1.c: Cannot open: Input/output error 2024-03-18T01:34:12.817 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sem_post/1-1.c: Cannot open: Input/output error 2024-03-18T01:34:12.817 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sem_post/8-1.c: Cannot open: Input/output error 2024-03-18T01:34:12.817 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sem_post/assertions.xml: Cannot open: Input/output error 2024-03-18T01:34:12.817 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sem_post/4-1.c: Cannot open: Input/output error 2024-03-18T01:34:12.818 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sem_post/cln.sh: Cannot open: Input/output error 2024-03-18T01:34:12.818 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/munlock: Cannot mkdir: Input/output error 2024-03-18T01:34:12.818 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/munlock/9-1.sh: Cannot open: Input/output error 2024-03-18T01:34:12.818 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/munlock/10-1.c: Cannot open: Input/output error 2024-03-18T01:34:12.819 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/munlock/coverage.txt: Cannot open: Input/output error 2024-03-18T01:34:12.819 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/munlock/assertions.xml: Cannot open: Input/output error 2024-03-18T01:34:12.819 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/munlock/7-1.c: Cannot open: Input/output error 2024-03-18T01:34:12.819 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/munlock/2-1.sh: Cannot open: Input/output error 2024-03-18T01:34:12.820 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/munlock/11-1.c: Cannot open: Input/output error 2024-03-18T01:34:12.820 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/munmap: Cannot mkdir: Input/output error 2024-03-18T01:34:12.820 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/munmap/3-1.c: Cannot open: Input/output error 2024-03-18T01:34:12.820 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/munmap/9-1.c: Cannot open: Input/output error 2024-03-18T01:34:12.821 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/munmap/coverage.txt: Cannot open: Input/output error 2024-03-18T01:34:12.821 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/munmap/1-2.c: Cannot open: Input/output error 2024-03-18T01:34:12.821 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/munmap/2-1.c: Cannot open: Input/output error 2024-03-18T01:34:12.821 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/munmap/1-1.c: Cannot open: Input/output error 2024-03-18T01:34:12.822 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/munmap/8-1.c: Cannot open: Input/output error 2024-03-18T01:34:12.822 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/munmap/7-1.sh: Cannot open: Input/output error 2024-03-18T01:34:12.822 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/munmap/assertions.xml: Cannot open: Input/output error 2024-03-18T01:34:12.822 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/munmap/10-1.sh: Cannot open: Input/output error 2024-03-18T01:34:12.823 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/munmap/4-1.c: Cannot open: Input/output error 2024-03-18T01:34:12.823 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_rwlockattr_setpshared: Cannot mkdir: Input/output error 2024-03-18T01:34:12.823 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_rwlockattr_setpshared/coverage.txt: Cannot open: Input/output error 2024-03-18T01:34:12.823 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_rwlockattr_setpshared/1-1.c: Cannot open: Input/output error 2024-03-18T01:34:12.824 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_rwlockattr_setpshared/assertions.xml: Cannot open: Input/output error 2024-03-18T01:34:12.824 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_rwlockattr_setpshared/2-1.sh: Cannot open: Input/output error 2024-03-18T01:34:12.824 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_sigmask: Cannot mkdir: Input/output error 2024-03-18T01:34:12.824 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_sigmask/8-3.c: Cannot open: Input/output error 2024-03-18T01:34:12.825 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_sigmask/9-1.c: Cannot open: Input/output error 2024-03-18T01:34:12.825 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_sigmask/10-1.c: Cannot open: Input/output error 2024-03-18T01:34:12.825 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_sigmask/coverage.txt: Cannot open: Input/output error 2024-03-18T01:34:12.825 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_sigmask/18-1.c: Cannot open: Input/output error 2024-03-18T01:34:12.826 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_sigmask/testfrmw.c: Cannot open: Input/output error 2024-03-18T01:34:12.826 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_sigmask/testfrmw.h: Cannot open: Input/output error 2024-03-18T01:34:12.826 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_sigmask/14-1.c: Cannot open: Input/output error 2024-03-18T01:34:12.827 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_sigmask/5-1.c: Cannot open: Input/output error 2024-03-18T01:34:12.827 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_sigmask/6-1.c: Cannot open: Input/output error 2024-03-18T01:34:12.827 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_sigmask/15-1.c: Cannot open: Input/output error 2024-03-18T01:34:12.827 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_sigmask/8-1.c: Cannot open: Input/output error 2024-03-18T01:34:12.828 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_sigmask/assertions.xml: Cannot open: Input/output error 2024-03-18T01:34:12.828 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_sigmask/12-1.c: Cannot open: Input/output error 2024-03-18T01:34:12.828 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_sigmask/7-1.c: Cannot open: Input/output error 2024-03-18T01:34:12.828 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_sigmask/4-1.c: Cannot open: Input/output error 2024-03-18T01:34:12.829 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_sigmask/8-2.c: Cannot open: Input/output error 2024-03-18T01:34:12.829 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_sigmask/16-1.c: Cannot open: Input/output error 2024-03-18T01:34:12.829 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mq_timedreceive: Cannot mkdir: Input/output error 2024-03-18T01:34:12.830 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mq_timedreceive/5-3.c: Cannot open: Input/output error 2024-03-18T01:34:12.830 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mq_timedreceive/10-1.c: Cannot open: Input/output error 2024-03-18T01:34:12.830 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mq_timedreceive/17-3.c: Cannot open: Input/output error 2024-03-18T01:34:12.830 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mq_timedreceive/coverage.txt: Cannot open: Input/output error 2024-03-18T01:34:12.831 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mq_timedreceive/18-1.c: Cannot open: Input/output error 2024-03-18T01:34:12.831 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mq_timedreceive/17-2.c: Cannot open: Input/output error 2024-03-18T01:34:12.831 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mq_timedreceive/14-1.c: Cannot open: Input/output error 2024-03-18T01:34:12.831 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mq_timedreceive/2-1.c: Cannot open: Input/output error 2024-03-18T01:34:12.831 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mq_timedreceive/5-1.c: Cannot open: Input/output error 2024-03-18T01:34:12.832 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mq_timedreceive/10-2.c: Cannot open: Input/output error 2024-03-18T01:34:12.832 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mq_timedreceive/1-1.c: Cannot open: Input/output error 2024-03-18T01:34:12.832 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mq_timedreceive/5-2.c: Cannot open: Input/output error 2024-03-18T01:34:12.832 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mq_timedreceive/15-1.c: Cannot open: Input/output error 2024-03-18T01:34:12.832 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mq_timedreceive/8-1.c: Cannot open: Input/output error 2024-03-18T01:34:12.832 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mq_timedreceive/12-1.sh: Cannot open: Input/output error 2024-03-18T01:34:12.833 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mq_timedreceive/assertions.xml: Cannot open: Input/output error 2024-03-18T01:34:12.833 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mq_timedreceive/7-1.c: Cannot open: Input/output error 2024-03-18T01:34:12.833 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mq_timedreceive/13-1.c: Cannot open: Input/output error 2024-03-18T01:34:12.833 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mq_timedreceive/4-1.sh: Cannot open: Input/output error 2024-03-18T01:34:12.833 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mq_timedreceive/16-1.sh: Cannot open: Input/output error 2024-03-18T01:34:12.834 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mq_timedreceive/18-2.c: Cannot open: Input/output error 2024-03-18T01:34:12.834 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mq_timedreceive/11-1.c: Cannot open: Input/output error 2024-03-18T01:34:12.834 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mq_timedreceive/17-1.c: Cannot open: Input/output error 2024-03-18T01:34:12.834 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mq_timedreceive/speculative: Cannot mkdir: Input/output error 2024-03-18T01:34:12.834 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mq_timedreceive/speculative/10-2.c: Cannot open: Input/output error 2024-03-18T01:34:12.835 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sem_wait: Cannot mkdir: Input/output error 2024-03-18T01:34:12.835 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sem_wait/3-1.c: Cannot open: Input/output error 2024-03-18T01:34:12.835 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sem_wait/coverage.txt: Cannot open: Input/output error 2024-03-18T01:34:12.835 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sem_wait/1-2.c: Cannot open: Input/output error 2024-03-18T01:34:12.836 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sem_wait/testfrmw.c: Cannot open: Input/output error 2024-03-18T01:34:12.836 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sem_wait/testfrmw.h: Cannot open: Input/output error 2024-03-18T01:34:12.836 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sem_wait/5-1.c: Cannot open: Input/output error 2024-03-18T01:34:12.836 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sem_wait/1-1.c: Cannot open: Input/output error 2024-03-18T01:34:12.836 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sem_wait/assertions.xml: Cannot open: Input/output error 2024-03-18T01:34:12.836 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sem_wait/12-1.c: Cannot open: Input/output error 2024-03-18T01:34:12.836 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sem_wait/7-1.c: Cannot open: Input/output error 2024-03-18T01:34:12.836 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sem_wait/13-1.c: Cannot open: Input/output error 2024-03-18T01:34:12.837 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sem_wait/11-1.c: Cannot open: Input/output error 2024-03-18T01:34:12.837 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sem_wait/cln.sh: Cannot open: Input/output error 2024-03-18T01:34:12.837 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigfillset: Cannot mkdir: Input/output error 2024-03-18T01:34:12.837 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigfillset/coverage.txt: Cannot open: Input/output error 2024-03-18T01:34:12.837 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigfillset/2-1.c: Cannot open: Input/output error 2024-03-18T01:34:12.837 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigfillset/1-1.c: Cannot open: Input/output error 2024-03-18T01:34:12.837 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigfillset/assertions.xml: Cannot open: Input/output error 2024-03-18T01:34:12.837 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sched_setscheduler: Cannot mkdir: Input/output error 2024-03-18T01:34:12.837 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sched_setscheduler/9-1.c: Cannot open: Input/output error 2024-03-18T01:34:12.838 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sched_setscheduler/20-1.c: Cannot open: Input/output error 2024-03-18T01:34:12.838 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sched_setscheduler/10-1.c: Cannot open: Input/output error 2024-03-18T01:34:12.838 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sched_setscheduler/19-3.c: Cannot open: Input/output error 2024-03-18T01:34:12.838 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sched_setscheduler/19-1.c: Cannot open: Input/output error 2024-03-18T01:34:12.838 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sched_setscheduler/22-2.c: Cannot open: Input/output error 2024-03-18T01:34:12.839 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sched_setscheduler/17-3.c: Cannot open: Input/output error 2024-03-18T01:34:12.839 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sched_setscheduler/coverage.txt: Cannot open: Input/output error 2024-03-18T01:34:12.839 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sched_setscheduler/17-5.c: Cannot open: Input/output error 2024-03-18T01:34:12.839 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sched_setscheduler/17-2.c: Cannot open: Input/output error 2024-03-18T01:34:12.839 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sched_setscheduler/19-2.c: Cannot open: Input/output error 2024-03-18T01:34:12.839 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sched_setscheduler/17-4.c: Cannot open: Input/output error 2024-03-18T01:34:12.839 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sched_setscheduler/17-7.c: Cannot open: Input/output error 2024-03-18T01:34:12.840 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sched_setscheduler/8-1.sh: Cannot open: Input/output error 2024-03-18T01:34:12.840 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sched_setscheduler/14-1.c: Cannot open: Input/output error 2024-03-18T01:34:12.840 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sched_setscheduler/2-1.c: Cannot open: Input/output error 2024-03-18T01:34:12.840 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sched_setscheduler/5-1.c: Cannot open: Input/output error 2024-03-18T01:34:12.840 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sched_setscheduler/6-1.c: Cannot open: Input/output error 2024-03-18T01:34:12.840 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sched_setscheduler/1-1.c: Cannot open: Input/output error 2024-03-18T01:34:12.840 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sched_setscheduler/19-4.c: Cannot open: Input/output error 2024-03-18T01:34:12.840 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sched_setscheduler/18-1.sh: Cannot open: Input/output error 2024-03-18T01:34:12.841 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sched_setscheduler/15-1.c: Cannot open: Input/output error 2024-03-18T01:34:12.841 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sched_setscheduler/assertions.xml: Cannot open: Input/output error 2024-03-18T01:34:12.841 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sched_setscheduler/15-2.c: Cannot open: Input/output error 2024-03-18T01:34:12.841 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sched_setscheduler/3-1.sh: Cannot open: Input/output error 2024-03-18T01:34:12.841 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sched_setscheduler/12-1.c: Cannot open: Input/output error 2024-03-18T01:34:12.841 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sched_setscheduler/7-1.c: Cannot open: Input/output error 2024-03-18T01:34:12.841 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sched_setscheduler/21-1.c: Cannot open: Input/output error 2024-03-18T01:34:12.841 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sched_setscheduler/17-6.c: Cannot open: Input/output error 2024-03-18T01:34:12.841 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sched_setscheduler/13-1.c: Cannot open: Input/output error 2024-03-18T01:34:12.842 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sched_setscheduler/22-1.c: Cannot open: Input/output error 2024-03-18T01:34:12.842 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sched_setscheduler/4-1.c: Cannot open: Input/output error 2024-03-18T01:34:12.842 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sched_setscheduler/19-5.c: Cannot open: Input/output error 2024-03-18T01:34:12.842 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sched_setscheduler/16-1.c: Cannot open: Input/output error 2024-03-18T01:34:12.842 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sched_setscheduler/11-1.c: Cannot open: Input/output error 2024-03-18T01:34:12.842 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sched_setscheduler/17-1.c: Cannot open: Input/output error 2024-03-18T01:34:12.842 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sem_getvalue: Cannot mkdir: Input/output error 2024-03-18T01:34:12.842 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sem_getvalue/testfrmw.c: Cannot open: Input/output error 2024-03-18T01:34:12.842 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sem_getvalue/testfrmw.h: Cannot open: Input/output error 2024-03-18T01:34:12.843 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sem_getvalue/2-1.c: Cannot open: Input/output error 2024-03-18T01:34:12.843 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sem_getvalue/5-1.c: Cannot open: Input/output error 2024-03-18T01:34:12.843 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sem_getvalue/1-1.c: Cannot open: Input/output error 2024-03-18T01:34:12.843 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sem_getvalue/coverage.c: Cannot open: Input/output error 2024-03-18T01:34:12.843 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sem_getvalue/assertions.xml: Cannot open: Input/output error 2024-03-18T01:34:12.843 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sem_getvalue/2-2.c: Cannot open: Input/output error 2024-03-18T01:34:12.843 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sem_getvalue/4-1.c: Cannot open: Input/output error 2024-03-18T01:34:12.843 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sem_getvalue/cln.sh: Cannot open: Input/output error 2024-03-18T01:34:12.843 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mq_receive: Cannot mkdir: Input/output error 2024-03-18T01:34:12.844 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mq_receive/9-1.sh: Cannot open: Input/output error 2024-03-18T01:34:12.844 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mq_receive/10-1.c: Cannot open: Input/output error 2024-03-18T01:34:12.844 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mq_receive/coverage.txt: Cannot open: Input/output error 2024-03-18T01:34:12.844 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mq_receive/2-1.c: Cannot open: Input/output error 2024-03-18T01:34:12.844 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mq_receive/5-1.c: Cannot open: Input/output error 2024-03-18T01:34:12.844 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mq_receive/1-1.c: Cannot open: Input/output error 2024-03-18T01:34:12.844 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mq_receive/8-1.c: Cannot open: Input/output error 2024-03-18T01:34:12.844 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mq_receive/assertions.xml: Cannot open: Input/output error 2024-03-18T01:34:12.845 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mq_receive/12-1.c: Cannot open: Input/output error 2024-03-18T01:34:12.845 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mq_receive/7-1.c: Cannot open: Input/output error 2024-03-18T01:34:12.845 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mq_receive/13-1.c: Cannot open: Input/output error 2024-03-18T01:34:12.845 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mq_receive/4-1.sh: Cannot open: Input/output error 2024-03-18T01:34:12.845 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mq_receive/11-2.c: Cannot open: Input/output error 2024-03-18T01:34:12.845 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mq_receive/11-1.c: Cannot open: Input/output error 2024-03-18T01:34:12.845 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_mutexattr_init: Cannot mkdir: Input/output error 2024-03-18T01:34:12.845 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_mutexattr_init/3-1.c: Cannot open: Input/output error 2024-03-18T01:34:12.845 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_mutexattr_init/coverage.txt: Cannot open: Input/output error 2024-03-18T01:34:12.846 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_mutexattr_init/1-1.c: Cannot open: Input/output error 2024-03-18T01:34:12.846 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_mutexattr_init/assertions.xml: Cannot open: Input/output error 2024-03-18T01:34:12.846 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sched_rr_get_interval: Cannot mkdir: Input/output error 2024-03-18T01:34:12.846 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sched_rr_get_interval/3-1.c: Cannot open: Input/output error 2024-03-18T01:34:12.846 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sched_rr_get_interval/coverage.txt: Cannot open: Input/output error 2024-03-18T01:34:12.846 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sched_rr_get_interval/2-1.c: Cannot open: Input/output error 2024-03-18T01:34:12.846 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sched_rr_get_interval/1-1.c: Cannot open: Input/output error 2024-03-18T01:34:12.846 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sched_rr_get_interval/assertions.xml: Cannot open: Input/output error 2024-03-18T01:34:12.847 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sched_rr_get_interval/4-1.sh: Cannot open: Input/output error 2024-03-18T01:34:12.847 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sched_rr_get_interval/speculative: Cannot mkdir: Input/output error 2024-03-18T01:34:12.847 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sched_rr_get_interval/speculative/5-1.c: Cannot open: Input/output error 2024-03-18T01:34:12.847 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_rwlockattr_getpshared: Cannot mkdir: Input/output error 2024-03-18T01:34:12.847 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_rwlockattr_getpshared/coverage.txt: Cannot open: Input/output error 2024-03-18T01:34:12.847 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_rwlockattr_getpshared/5-1.sh: Cannot open: Input/output error 2024-03-18T01:34:12.847 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_rwlockattr_getpshared/2-1.c: Cannot open: Input/output error 2024-03-18T01:34:12.847 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_rwlockattr_getpshared/1-1.c: Cannot open: Input/output error 2024-03-18T01:34:12.847 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_rwlockattr_getpshared/assertions.xml: Cannot open: Input/output error 2024-03-18T01:34:12.848 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_rwlockattr_getpshared/4-1.c: Cannot open: Input/output error 2024-03-18T01:34:12.848 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigpending: Cannot mkdir: Input/output error 2024-03-18T01:34:12.848 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigpending/coverage.txt: Cannot open: Input/output error 2024-03-18T01:34:12.848 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigpending/1-2.c: Cannot open: Input/output error 2024-03-18T01:34:12.848 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigpending/2-1.c: Cannot open: Input/output error 2024-03-18T01:34:12.848 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigpending/1-1.c: Cannot open: Input/output error 2024-03-18T01:34:12.848 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigpending/assertions.xml: Cannot open: Input/output error 2024-03-18T01:34:12.848 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigpending/1-3.c: Cannot open: Input/output error 2024-03-18T01:34:12.849 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_attr_getstackaddr: Cannot mkdir: Input/output error 2024-03-18T01:34:12.849 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_attr_getstackaddr/coverage.txt: Cannot open: Input/output error 2024-03-18T01:34:12.849 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_attr_getstackaddr/assertions.xml: Cannot open: Input/output error 2024-03-18T01:34:12.849 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_exit: Cannot mkdir: Input/output error 2024-03-18T01:34:12.849 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_exit/3-1.c: Cannot open: Input/output error 2024-03-18T01:34:12.849 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_exit/6-2.c: Cannot open: Input/output error 2024-03-18T01:34:12.849 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_exit/coverage.txt: Cannot open: Input/output error 2024-03-18T01:34:12.849 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_exit/1-2.c: Cannot open: Input/output error 2024-03-18T01:34:12.849 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_exit/testfrmw.c: Cannot open: Input/output error 2024-03-18T01:34:12.850 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_exit/testfrmw.h: Cannot open: Input/output error 2024-03-18T01:34:12.850 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_exit/2-1.c: Cannot open: Input/output error 2024-03-18T01:34:12.850 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_exit/5-1.c: Cannot open: Input/output error 2024-03-18T01:34:12.850 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_exit/6-1.c: Cannot open: Input/output error 2024-03-18T01:34:12.850 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_exit/1-1.c: Cannot open: Input/output error 2024-03-18T01:34:12.850 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_exit/assertions.xml: Cannot open: Input/output error 2024-03-18T01:34:12.850 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_exit/2-2.c: Cannot open: Input/output error 2024-03-18T01:34:12.850 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_exit/4-1.c: Cannot open: Input/output error 2024-03-18T01:34:12.850 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_exit/threads_scenarii.c: Cannot open: Input/output error 2024-03-18T01:34:12.851 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_exit/3-2.c: Cannot open: Input/output error 2024-03-18T01:34:12.851 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_mutex_unlock: Cannot mkdir: Input/output error 2024-03-18T01:34:12.851 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_mutex_unlock/3-1.c: Cannot open: Input/output error 2024-03-18T01:34:12.851 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_mutex_unlock/coverage.txt: Cannot open: Input/output error 2024-03-18T01:34:12.851 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_mutex_unlock/testfrmw.c: Cannot open: Input/output error 2024-03-18T01:34:12.851 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_mutex_unlock/testfrmw.h: Cannot open: Input/output error 2024-03-18T01:34:12.851 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_mutex_unlock/2-1.c: Cannot open: Input/output error 2024-03-18T01:34:12.851 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_mutex_unlock/5-1.c: Cannot open: Input/output error 2024-03-18T01:34:12.851 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_mutex_unlock/1-1.c: Cannot open: Input/output error 2024-03-18T01:34:12.852 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_mutex_unlock/5-2.c: Cannot open: Input/output error 2024-03-18T01:34:12.852 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_mutex_unlock/assertions.xml: Cannot open: Input/output error 2024-03-18T01:34:12.852 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_rwlockattr_destroy: Cannot mkdir: Input/output error 2024-03-18T01:34:12.852 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_rwlockattr_destroy/coverage.txt: Cannot open: Input/output error 2024-03-18T01:34:12.852 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_rwlockattr_destroy/2-1.c: Cannot open: Input/output error 2024-03-18T01:34:12.852 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_rwlockattr_destroy/1-1.c: Cannot open: Input/output error 2024-03-18T01:34:12.852 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_rwlockattr_destroy/assertions.xml: Cannot open: Input/output error 2024-03-18T01:34:12.852 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_rwlockattr_destroy/3-1.sh: Cannot open: Input/output error 2024-03-18T01:34:12.852 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mq_notify: Cannot mkdir: Input/output error 2024-03-18T01:34:12.853 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mq_notify/3-1.c: Cannot open: Input/output error 2024-03-18T01:34:12.853 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mq_notify/9-1.c: Cannot open: Input/output error 2024-03-18T01:34:12.853 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mq_notify/coverage.txt: Cannot open: Input/output error 2024-03-18T01:34:12.853 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mq_notify/6-1.sh: Cannot open: Input/output error 2024-03-18T01:34:12.853 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mq_notify/2-1.c: Cannot open: Input/output error 2024-03-18T01:34:12.853 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mq_notify/5-1.c: Cannot open: Input/output error 2024-03-18T01:34:12.853 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mq_notify/1-1.c: Cannot open: Input/output error 2024-03-18T01:34:12.853 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mq_notify/8-1.c: Cannot open: Input/output error 2024-03-18T01:34:12.853 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mq_notify/7-1.sh: Cannot open: Input/output error 2024-03-18T01:34:12.854 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mq_notify/assertions.xml: Cannot open: Input/output error 2024-03-18T01:34:12.854 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mq_notify/4-1.c: Cannot open: Input/output error 2024-03-18T01:34:12.854 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sched_yield: Cannot mkdir: Input/output error 2024-03-18T01:34:12.854 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sched_yield/coverage.txt: Cannot open: Input/output error 2024-03-18T01:34:12.854 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sched_yield/2-1.c: Cannot open: Input/output error 2024-03-18T01:34:12.854 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sched_yield/1-1.c: Cannot open: Input/output error 2024-03-18T01:34:12.854 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sched_yield/assertions.xml: Cannot open: Input/output error 2024-03-18T01:34:12.854 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sched_yield/3-1.sh: Cannot open: Input/output error 2024-03-18T01:34:12.854 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/aio_fsync: Cannot mkdir: Input/output error 2024-03-18T01:34:12.855 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/aio_fsync/8-3.c: Cannot open: Input/output error 2024-03-18T01:34:12.855 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/aio_fsync/3-1.c: Cannot open: Input/output error 2024-03-18T01:34:12.855 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/aio_fsync/9-1.c: Cannot open: Input/output error 2024-03-18T01:34:12.855 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/aio_fsync/4-2.c: Cannot open: Input/output error 2024-03-18T01:34:12.855 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/aio_fsync/10-1.c: Cannot open: Input/output error 2024-03-18T01:34:12.855 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/aio_fsync/coverage.txt: Cannot open: Input/output error 2024-03-18T01:34:12.855 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/aio_fsync/8-4.c: Cannot open: Input/output error 2024-03-18T01:34:12.855 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/aio_fsync/14-1.c: Cannot open: Input/output error 2024-03-18T01:34:12.855 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/aio_fsync/2-1.c: Cannot open: Input/output error 2024-03-18T01:34:12.856 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/aio_fsync/5-1.c: Cannot open: Input/output error 2024-03-18T01:34:12.856 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/aio_fsync/6-1.c: Cannot open: Input/output error 2024-03-18T01:34:12.856 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/aio_fsync/1-1.c: Cannot open: Input/output error 2024-03-18T01:34:12.856 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/aio_fsync/8-1.c: Cannot open: Input/output error 2024-03-18T01:34:12.856 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/aio_fsync/assertions.xml: Cannot open: Input/output error 2024-03-18T01:34:12.856 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/aio_fsync/12-1.c: Cannot open: Input/output error 2024-03-18T01:34:12.856 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/aio_fsync/7-1.c: Cannot open: Input/output error 2024-03-18T01:34:12.856 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/aio_fsync/13-1.c: Cannot open: Input/output error 2024-03-18T01:34:12.856 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/aio_fsync/4-1.c: Cannot open: Input/output error 2024-03-18T01:34:12.856 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/aio_fsync/8-2.c: Cannot open: Input/output error 2024-03-18T01:34:12.857 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/aio_fsync/11-1.c: Cannot open: Input/output error 2024-03-18T01:34:12.857 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_attr_setstack: Cannot mkdir: Input/output error 2024-03-18T01:34:12.857 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_attr_setstack/coverage.txt: Cannot open: Input/output error 2024-03-18T01:34:12.857 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_attr_setstack/5-1.sh: Cannot open: Input/output error 2024-03-18T01:34:12.857 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_attr_setstack/2-1.c: Cannot open: Input/output error 2024-03-18T01:34:12.857 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_attr_setstack/6-1.c: Cannot open: Input/output error 2024-03-18T01:34:12.857 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_attr_setstack/1-1.c: Cannot open: Input/output error 2024-03-18T01:34:12.857 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_attr_setstack/assertions.xml: Cannot open: Input/output error 2024-03-18T01:34:12.857 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_attr_setstack/3-1.sh: Cannot open: Input/output error 2024-03-18T01:34:12.858 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_attr_setstack/7-1.c: Cannot open: Input/output error 2024-03-18T01:34:12.858 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_attr_setstack/4-1.c: Cannot open: Input/output error 2024-03-18T01:34:12.858 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_condattr_init: Cannot mkdir: Input/output error 2024-03-18T01:34:12.858 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_condattr_init/3-1.c: Cannot open: Input/output error 2024-03-18T01:34:12.858 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_condattr_init/coverage.txt: Cannot open: Input/output error 2024-03-18T01:34:12.858 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_condattr_init/1-1.c: Cannot open: Input/output error 2024-03-18T01:34:12.858 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_condattr_init/assertions.xml: Cannot open: Input/output error 2024-03-18T01:34:12.858 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mktime: Cannot mkdir: Input/output error 2024-03-18T01:34:12.859 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mktime/coverage.txt: Cannot open: Input/output error 2024-03-18T01:34:12.859 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mktime/1-1.c: Cannot open: Input/output error 2024-03-18T01:34:12.859 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mktime/assertions.xml: Cannot open: Input/output error 2024-03-18T01:34:12.859 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mq_send: Cannot mkdir: Input/output error 2024-03-18T01:34:12.859 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mq_send/3-1.c: Cannot open: Input/output error 2024-03-18T01:34:12.859 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mq_send/9-1.c: Cannot open: Input/output error 2024-03-18T01:34:12.859 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mq_send/4-2.c: Cannot open: Input/output error 2024-03-18T01:34:12.859 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mq_send/10-1.c: Cannot open: Input/output error 2024-03-18T01:34:12.859 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mq_send/coverage.txt: Cannot open: Input/output error 2024-03-18T01:34:12.860 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mq_send/14-1.c: Cannot open: Input/output error 2024-03-18T01:34:12.860 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mq_send/2-1.c: Cannot open: Input/output error 2024-03-18T01:34:12.860 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mq_send/5-1.c: Cannot open: Input/output error 2024-03-18T01:34:12.860 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mq_send/6-1.c: Cannot open: Input/output error 2024-03-18T01:34:12.860 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mq_send/1-1.c: Cannot open: Input/output error 2024-03-18T01:34:12.860 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mq_send/5-2.c: Cannot open: Input/output error 2024-03-18T01:34:12.860 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mq_send/8-1.c: Cannot open: Input/output error 2024-03-18T01:34:12.860 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mq_send/assertions.xml: Cannot open: Input/output error 2024-03-18T01:34:12.860 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mq_send/12-1.c: Cannot open: Input/output error 2024-03-18T01:34:12.861 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mq_send/7-1.c: Cannot open: Input/output error 2024-03-18T01:34:12.861 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mq_send/4-3.c: Cannot open: Input/output error 2024-03-18T01:34:12.861 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mq_send/13-1.c: Cannot open: Input/output error 2024-03-18T01:34:12.861 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mq_send/11-2.c: Cannot open: Input/output error 2024-03-18T01:34:12.861 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mq_send/4-1.c: Cannot open: Input/output error 2024-03-18T01:34:12.861 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mq_send/11-1.c: Cannot open: Input/output error 2024-03-18T01:34:12.861 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mq_send/3-2.c: Cannot open: Input/output error 2024-03-18T01:34:12.861 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/aio_return: Cannot mkdir: Input/output error 2024-03-18T01:34:12.861 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/aio_return/3-1.c: Cannot open: Input/output error 2024-03-18T01:34:12.861 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/aio_return/coverage.txt: Cannot open: Input/output error 2024-03-18T01:34:12.862 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/aio_return/2-1.c: Cannot open: Input/output error 2024-03-18T01:34:12.862 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/aio_return/1-1.c: Cannot open: Input/output error 2024-03-18T01:34:12.862 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/aio_return/assertions.xml: Cannot open: Input/output error 2024-03-18T01:34:12.862 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/aio_return/4-1.c: Cannot open: Input/output error 2024-03-18T01:34:12.862 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/aio_return/3-2.c: Cannot open: Input/output error 2024-03-18T01:34:12.862 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigset: Cannot mkdir: Input/output error 2024-03-18T01:34:12.862 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigset/3-1.c: Cannot open: Input/output error 2024-03-18T01:34:12.862 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigset/9-1.c: Cannot open: Input/output error 2024-03-18T01:34:12.862 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigset/10-1.c: Cannot open: Input/output error 2024-03-18T01:34:12.863 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigset/coverage.txt: Cannot open: Input/output error 2024-03-18T01:34:12.863 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigset/2-1.c: Cannot open: Input/output error 2024-03-18T01:34:12.863 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigset/5-1.c: Cannot open: Input/output error 2024-03-18T01:34:12.863 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigset/6-1.c: Cannot open: Input/output error 2024-03-18T01:34:12.863 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigset/1-1.c: Cannot open: Input/output error 2024-03-18T01:34:12.863 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigset/8-1.c: Cannot open: Input/output error 2024-03-18T01:34:12.863 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigset/assertions.xml: Cannot open: Input/output error 2024-03-18T01:34:12.863 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigset/7-1.c: Cannot open: Input/output error 2024-03-18T01:34:12.863 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigset/4-1.c: Cannot open: Input/output error 2024-03-18T01:34:12.864 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_attr_init: Cannot mkdir: Input/output error 2024-03-18T01:34:12.864 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_attr_init/3-1.c: Cannot open: Input/output error 2024-03-18T01:34:12.864 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_attr_init/coverage.txt: Cannot open: Input/output error 2024-03-18T01:34:12.864 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_attr_init/2-1.c: Cannot open: Input/output error 2024-03-18T01:34:12.864 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_attr_init/1-1.c: Cannot open: Input/output error 2024-03-18T01:34:12.864 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_attr_init/assertions.xml: Cannot open: Input/output error 2024-03-18T01:34:12.864 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_attr_init/4-1.c: Cannot open: Input/output error 2024-03-18T01:34:12.864 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_mutex_trylock: Cannot mkdir: Input/output error 2024-03-18T01:34:12.864 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_mutex_trylock/3-1.c: Cannot open: Input/output error 2024-03-18T01:34:12.865 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_mutex_trylock/4-2.c: Cannot open: Input/output error 2024-03-18T01:34:12.865 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_mutex_trylock/coverage.txt: Cannot open: Input/output error 2024-03-18T01:34:12.865 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_mutex_trylock/1-2.c: Cannot open: Input/output error 2024-03-18T01:34:12.865 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_mutex_trylock/testfrmw.c: Cannot open: Input/output error 2024-03-18T01:34:12.865 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_mutex_trylock/testfrmw.h: Cannot open: Input/output error 2024-03-18T01:34:12.865 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_mutex_trylock/2-1.c: Cannot open: Input/output error 2024-03-18T01:34:12.865 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_mutex_trylock/1-1.c: Cannot open: Input/output error 2024-03-18T01:34:12.865 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_mutex_trylock/assertions.xml: Cannot open: Input/output error 2024-03-18T01:34:12.865 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_mutex_trylock/4-3.c: Cannot open: Input/output error 2024-03-18T01:34:12.866 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_mutex_trylock/4-1.c: Cannot open: Input/output error 2024-03-18T01:34:12.866 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mq_getattr: Cannot mkdir: Input/output error 2024-03-18T01:34:12.866 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mq_getattr/3-1.c: Cannot open: Input/output error 2024-03-18T01:34:12.866 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mq_getattr/coverage.txt: Cannot open: Input/output error 2024-03-18T01:34:12.866 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mq_getattr/5-1.sh: Cannot open: Input/output error 2024-03-18T01:34:12.866 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mq_getattr/6-1.sh: Cannot open: Input/output error 2024-03-18T01:34:12.866 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mq_getattr/2-1.c: Cannot open: Input/output error 2024-03-18T01:34:12.866 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mq_getattr/1-1.sh: Cannot open: Input/output error 2024-03-18T01:34:12.866 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mq_getattr/assertions.xml: Cannot open: Input/output error 2024-03-18T01:34:12.867 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mq_getattr/2-2.c: Cannot open: Input/output error 2024-03-18T01:34:12.867 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mq_getattr/4-1.c: Cannot open: Input/output error 2024-03-18T01:34:12.867 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mq_getattr/speculative: Cannot mkdir: Input/output error 2024-03-18T01:34:12.867 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mq_getattr/speculative/7-1.c: Cannot open: Input/output error 2024-03-18T01:34:12.867 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_join: Cannot mkdir: Input/output error 2024-03-18T01:34:12.867 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_join/3-1.c: Cannot open: Input/output error 2024-03-18T01:34:12.867 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_join/6-2.c: Cannot open: Input/output error 2024-03-18T01:34:12.867 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_join/coverage.txt: Cannot open: Input/output error 2024-03-18T01:34:12.867 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_join/1-2.c: Cannot open: Input/output error 2024-03-18T01:34:12.868 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_join/testfrmw.c: Cannot open: Input/output error 2024-03-18T01:34:12.868 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_join/testfrmw.h: Cannot open: Input/output error 2024-03-18T01:34:12.868 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_join/2-1.c: Cannot open: Input/output error 2024-03-18T01:34:12.868 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_join/5-1.c: Cannot open: Input/output error 2024-03-18T01:34:12.868 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_join/1-1.c: Cannot open: Input/output error 2024-03-18T01:34:12.868 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_join/assertions.xml: Cannot open: Input/output error 2024-03-18T01:34:12.868 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_join/4-1.c: Cannot open: Input/output error 2024-03-18T01:34:12.868 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_join/threads_scenarii.c: Cannot open: Input/output error 2024-03-18T01:34:12.868 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_join/6-3.c: Cannot open: Input/output error 2024-03-18T01:34:12.869 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_join/speculative: Cannot mkdir: Input/output error 2024-03-18T01:34:12.869 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_join/speculative/6-1.c: Cannot open: Input/output error 2024-03-18T01:34:12.869 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_attr_getstacksize: Cannot mkdir: Input/output error 2024-03-18T01:34:12.869 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_attr_getstacksize/coverage.txt: Cannot open: Input/output error 2024-03-18T01:34:12.869 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_attr_getstacksize/1-1.c: Cannot open: Input/output error 2024-03-18T01:34:12.869 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_attr_getstacksize/assertions.xml: Cannot open: Input/output error 2024-03-18T01:34:12.869 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_attr_getstacksize/2-1.sh: Cannot open: Input/output error 2024-03-18T01:34:12.869 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_attr_getinheritsched: Cannot mkdir: Input/output error 2024-03-18T01:34:12.869 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_attr_getinheritsched/coverage.txt: Cannot open: Input/output error 2024-03-18T01:34:12.870 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_attr_getinheritsched/1-1.c: Cannot open: Input/output error 2024-03-18T01:34:12.870 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_attr_getinheritsched/assertions.xml: Cannot open: Input/output error 2024-03-18T01:34:12.870 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_attr_getinheritsched/2-1.sh: Cannot open: Input/output error 2024-03-18T01:34:12.870 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigwait: Cannot mkdir: Input/output error 2024-03-18T01:34:12.870 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigwait/3-1.c: Cannot open: Input/output error 2024-03-18T01:34:12.870 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigwait/6-2.c: Cannot open: Input/output error 2024-03-18T01:34:12.870 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigwait/coverage.txt: Cannot open: Input/output error 2024-03-18T01:34:12.870 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigwait/testfrmw.c: Cannot open: Input/output error 2024-03-18T01:34:12.871 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigwait/testfrmw.h: Cannot open: Input/output error 2024-03-18T01:34:12.871 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigwait/2-1.c: Cannot open: Input/output error 2024-03-18T01:34:12.871 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigwait/6-1.c: Cannot open: Input/output error 2024-03-18T01:34:12.871 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigwait/1-1.c: Cannot open: Input/output error 2024-03-18T01:34:12.871 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigwait/8-1.c: Cannot open: Input/output error 2024-03-18T01:34:12.871 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigwait/assertions.xml: Cannot open: Input/output error 2024-03-18T01:34:12.871 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigwait/7-1.c: Cannot open: Input/output error 2024-03-18T01:34:12.871 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigwait/4-1.c: Cannot open: Input/output error 2024-03-18T01:34:12.871 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/strftime: Cannot mkdir: Input/output error 2024-03-18T01:34:12.872 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/strftime/3-1.c: Cannot open: Input/output error 2024-03-18T01:34:12.872 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/strftime/coverage.txt: Cannot open: Input/output error 2024-03-18T01:34:12.872 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/strftime/2-1.c: Cannot open: Input/output error 2024-03-18T01:34:12.872 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/strftime/1-1.c: Cannot open: Input/output error 2024-03-18T01:34:12.872 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/strftime/assertions.xml: Cannot open: Input/output error 2024-03-18T01:34:12.872 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sched_get_priority_max: Cannot mkdir: Input/output error 2024-03-18T01:34:12.872 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sched_get_priority_max/coverage.txt: Cannot open: Input/output error 2024-03-18T01:34:12.872 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sched_get_priority_max/1-2.c: Cannot open: Input/output error 2024-03-18T01:34:12.873 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sched_get_priority_max/2-1.c: Cannot open: Input/output error 2024-03-18T01:34:12.873 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sched_get_priority_max/1-1.c: Cannot open: Input/output error 2024-03-18T01:34:12.873 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sched_get_priority_max/assertions.xml: Cannot open: Input/output error 2024-03-18T01:34:12.873 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sched_get_priority_max/3-1.sh: Cannot open: Input/output error 2024-03-18T01:34:12.873 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sched_get_priority_max/1-3.c: Cannot open: Input/output error 2024-03-18T01:34:12.873 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sched_get_priority_max/1-4.c: Cannot open: Input/output error 2024-03-18T01:34:12.873 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/timer_delete: Cannot mkdir: Input/output error 2024-03-18T01:34:12.873 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/timer_delete/coverage.txt: Cannot open: Input/output error 2024-03-18T01:34:12.873 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/timer_delete/1-2.c: Cannot open: Input/output error 2024-03-18T01:34:12.873 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/timer_delete/1-1.c: Cannot open: Input/output error 2024-03-18T01:34:12.874 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/timer_delete/assertions.xml: Cannot open: Input/output error 2024-03-18T01:34:12.874 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/timer_delete/3-1.sh: Cannot open: Input/output error 2024-03-18T01:34:12.874 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/timer_delete/4-1.sh: Cannot open: Input/output error 2024-03-18T01:34:12.874 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/timer_delete/2-1.sh: Cannot open: Input/output error 2024-03-18T01:34:12.874 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/timer_delete/speculative: Cannot mkdir: Input/output error 2024-03-18T01:34:12.874 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/timer_delete/speculative/5-1.c: Cannot open: Input/output error 2024-03-18T01:34:12.874 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/timer_delete/speculative/5-2.c: Cannot open: Input/output error 2024-03-18T01:34:12.874 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/gmtime: Cannot mkdir: Input/output error 2024-03-18T01:34:12.875 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/gmtime/coverage.txt: Cannot open: Input/output error 2024-03-18T01:34:12.875 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/gmtime/2-1.c: Cannot open: Input/output error 2024-03-18T01:34:12.875 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/gmtime/1-1.c: Cannot open: Input/output error 2024-03-18T01:34:12.875 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/gmtime/assertions.xml: Cannot open: Input/output error 2024-03-18T01:34:12.875 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_condattr_destroy: Cannot mkdir: Input/output error 2024-03-18T01:34:12.875 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_condattr_destroy/3-1.c: Cannot open: Input/output error 2024-03-18T01:34:12.875 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_condattr_destroy/coverage.txt: Cannot open: Input/output error 2024-03-18T01:34:12.875 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_condattr_destroy/2-1.c: Cannot open: Input/output error 2024-03-18T01:34:12.875 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_condattr_destroy/1-1.c: Cannot open: Input/output error 2024-03-18T01:34:12.876 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_condattr_destroy/assertions.xml: Cannot open: Input/output error 2024-03-18T01:34:12.876 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_condattr_destroy/4-1.c: Cannot open: Input/output error 2024-03-18T01:34:12.876 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_rwlock_tryrdlock: Cannot mkdir: Input/output error 2024-03-18T01:34:12.876 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_rwlock_tryrdlock/coverage.txt: Cannot open: Input/output error 2024-03-18T01:34:12.876 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_rwlock_tryrdlock/1-1.c: Cannot open: Input/output error 2024-03-18T01:34:12.876 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_rwlock_tryrdlock/assertions.xml: Cannot open: Input/output error 2024-03-18T01:34:12.876 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_rwlock_tryrdlock/3-1.sh: Cannot open: Input/output error 2024-03-18T01:34:12.876 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_rwlock_tryrdlock/2-1.sh: Cannot open: Input/output error 2024-03-18T01:34:12.876 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sched_setparam: Cannot mkdir: Input/output error 2024-03-18T01:34:12.877 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sched_setparam/27-1.c: Cannot open: Input/output error 2024-03-18T01:34:12.877 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sched_setparam/3-1.c: Cannot open: Input/output error 2024-03-18T01:34:12.877 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sched_setparam/9-1.c: Cannot open: Input/output error 2024-03-18T01:34:12.877 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sched_setparam/20-1.c: Cannot open: Input/output error 2024-03-18T01:34:12.877 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sched_setparam/10-1.c: Cannot open: Input/output error 2024-03-18T01:34:12.877 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sched_setparam/23-1.c: Cannot open: Input/output error 2024-03-18T01:34:12.877 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sched_setparam/19-1.c: Cannot open: Input/output error 2024-03-18T01:34:12.877 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sched_setparam/coverage.txt: Cannot open: Input/output error 2024-03-18T01:34:12.877 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sched_setparam/18-1.c: Cannot open: Input/output error 2024-03-18T01:34:12.878 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sched_setparam/23-3.c: Cannot open: Input/output error 2024-03-18T01:34:12.878 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sched_setparam/25-4.c: Cannot open: Input/output error 2024-03-18T01:34:12.878 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sched_setparam/25-2.c: Cannot open: Input/output error 2024-03-18T01:34:12.878 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sched_setparam/14-1.c: Cannot open: Input/output error 2024-03-18T01:34:12.878 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sched_setparam/2-1.c: Cannot open: Input/output error 2024-03-18T01:34:12.878 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sched_setparam/5-1.c: Cannot open: Input/output error 2024-03-18T01:34:12.878 INFO:tasks.workunit.client.0.smithi029.stderr: 2024-03-18T01:34:12.878 INFO:tasks.workunit.client.0.smithi029.stderr:gzip: tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sched_setparam/6-1.c: Cannot open: Input/output error 2024-03-18T01:34:12.878 INFO:tasks.workunit.client.0.smithi029.stderr:stdin: Stale file handle 2024-03-18T01:34:12.879 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sched_setparam/1-1.c: Cannot open: Input/output error 2024-03-18T01:34:12.879 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sched_setparam/23-2.c: Cannot open: Input/output error 2024-03-18T01:34:12.879 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sched_setparam/15-1.c: Cannot open: Input/output error 2024-03-18T01:34:12.879 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sched_setparam/8-1.c: Cannot open: Input/output error 2024-03-18T01:34:12.879 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sched_setparam/24-1.sh: Cannot open: Input/output error 2024-03-18T01:34:12.879 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sched_setparam/assertions.xml: Cannot open: Input/output error 2024-03-18T01:34:12.879 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sched_setparam/12-1.c: Cannot open: Input/output error 2024-03-18T01:34:12.879 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sched_setparam/7-1.c: Cannot open: Input/output error 2024-03-18T01:34:12.879 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sched_setparam/21-1.c: Cannot open: Input/output error 2024-03-18T01:34:12.880 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sched_setparam/23-6.c: Cannot open: Input/output error 2024-03-18T01:34:12.880 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sched_setparam/23-7.c: Cannot open: Input/output error 2024-03-18T01:34:12.880 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sched_setparam/2-2.c: Cannot open: Input/output error 2024-03-18T01:34:12.880 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sched_setparam/25-3.c: Cannot open: Input/output error 2024-03-18T01:34:12.880 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sched_setparam/13-1.c: Cannot open: Input/output error 2024-03-18T01:34:12.880 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sched_setparam/23-4.c: Cannot open: Input/output error 2024-03-18T01:34:12.880 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sched_setparam/4-1.sh: Cannot open: Input/output error 2024-03-18T01:34:12.880 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sched_setparam/22-1.c: Cannot open: Input/output error 2024-03-18T01:34:12.880 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sched_setparam/26-1.c: Cannot open: Input/output error 2024-03-18T01:34:12.881 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sched_setparam/11-1.sh: Cannot open: Input/output error 2024-03-18T01:34:12.881 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sched_setparam/23-5.c: Cannot open: Input/output error 2024-03-18T01:34:12.881 INFO:tasks.workunit.client.0.smithi029.stderr:tar: Unexpected EOF in archive 2024-03-18T01:34:12.881 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces: Cannot utime: Input/output error 2024-03-18T01:34:12.881 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance: Cannot utime: Input/output error 2024-03-18T01:34:12.881 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite: Cannot utime: Input/output error 2024-03-18T01:34:12.881 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases: Cannot utime: Input/output error 2024-03-18T01:34:12.881 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231: Cannot utime: Input/output error 2024-03-18T01:34:12.881 INFO:tasks.workunit.client.0.smithi029.stderr:tar: Error is not recoverable: exiting now 2024-03-18T01:34:12.883 DEBUG:teuthology.orchestra.run:got remote process result: 2 2024-03-18T01:34:12.883 INFO:tasks.workunit:Stopping ['suites/fsstress.sh'] on client.0... 2024-03-18T01:34:12.883 DEBUG:teuthology.orchestra.run.smithi029:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-03-18T01:34:12.883 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi029.front.sepia.ceph.com 2024-03-18T01:34:12.884 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T01:34:12.884 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-18T01:34:12.884 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:34:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:34:12.568+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:34:12.885 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:34:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:34:12.568+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:34:12.885 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:34:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:34:12.568+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:34:12.885 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:34:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:34:12.569+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:34:12.885 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:34:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:34:12.626+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:34:12.885 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:34:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:34:12.626+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:34:12.885 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:34:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:34:12.626+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:34:12.885 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:34:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:34:12.627+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:34:12.951 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-03-18T01:34:12.951 DEBUG:teuthology.orchestra.run.smithi090:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-18T01:34:13.208 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:34:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:34:12.909+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:34:13.208 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:34:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:34:12.909+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:34:13.208 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:34:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:34:12.909+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:34:13.208 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:34:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:34:12.909+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:34:13.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:34:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:34:13.652+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:34:13.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:34:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:34:13.652+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:34:13.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:34:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:34:13.652+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:34:13.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:34:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:34:13.653+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:34:13.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:34:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:34:13.565+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:34:13.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:34:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:34:13.565+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:34:13.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:34:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:34:13.565+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:34:13.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:34:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:34:13.566+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:34:14.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:34:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:34:13.919+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:34:14.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:34:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:34:13.919+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:34:14.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:34:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:34:13.919+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:34:14.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:34:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:34:13.920+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:34:14.668 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:34:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:34:14.554+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:34:14.668 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:34:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:34:14.554+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:34:14.668 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:34:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:34:14.554+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:34:14.668 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:34:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:34:14.555+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:34:14.926 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:34:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:34:14.667+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:34:14.926 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:34:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:34:14.667+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:34:14.926 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:34:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:34:14.667+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:34:14.926 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:34:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:34:14.668+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:34:15.265 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:34:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:34:14.925+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:34:15.265 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:34:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:34:14.925+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:34:15.265 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:34:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:34:14.925+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:34:15.265 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:34:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:34:14.926+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:34:15.516 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:34:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:34:15.264+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 11284 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:34:15.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:34:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:34:15.662+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:34:15.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:34:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:34:15.662+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:34:15.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:34:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:34:15.662+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:34:15.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:34:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:34:15.663+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:34:15.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:34:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:34:15.515+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:34:15.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:34:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:34:15.515+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:34:15.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:34:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:34:15.515+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:34:15.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:34:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:34:15.516+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:34:16.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:34:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:34:15.963+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:34:16.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:34:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:34:15.963+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:34:16.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:34:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:34:15.963+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:34:16.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:34:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:34:15.963+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:34:16.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:34:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:34:16.679+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:34:16.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:34:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:34:16.679+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:34:16.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:34:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:34:16.679+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:34:16.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:34:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:34:16.680+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:34:16.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:34:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:34:16.498+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:34:16.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:34:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:34:16.498+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:34:16.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:34:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:34:16.498+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:34:16.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:34:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:34:16.499+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:34:17.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:34:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:34:16.972+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:34:17.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:34:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:34:16.972+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:34:17.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:34:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:34:16.972+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:34:17.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:34:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:34:16.972+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:34:17.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:34:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:34:17.711+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:34:17.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:34:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:34:17.711+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:34:17.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:34:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:34:17.711+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:34:17.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:34:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:34:17.712+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:34:17.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:34:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:34:17.515+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:34:17.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:34:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:34:17.515+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:34:17.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:34:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:34:17.515+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:34:17.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:34:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:34:17.516+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:34:18.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:34:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:34:17.945+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:34:18.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:34:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:34:17.945+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:34:18.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:34:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:34:17.945+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:34:18.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:34:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:34:17.946+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:34:18.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:34:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:34:18.667+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:34:18.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:34:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:34:18.667+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:34:18.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:34:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:34:18.667+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:34:18.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:34:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:34:18.668+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:34:18.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:34:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:34:18.512+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:34:18.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:34:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:34:18.512+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:34:18.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:34:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:34:18.512+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:34:18.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:34:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:34:18.512+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:34:19.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:34:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:34:18.926+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:34:19.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:34:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:34:18.926+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:34:19.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:34:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:34:18.926+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:34:19.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:34:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:34:18.927+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:34:19.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:34:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:34:19.661+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:34:19.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:34:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:34:19.661+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:34:19.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:34:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:34:19.661+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:34:19.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:34:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:34:19.662+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:34:19.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:34:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:34:19.539+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:34:19.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:34:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:34:19.539+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:34:19.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:34:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:34:19.539+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:34:19.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:34:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:34:19.540+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:34:20.266 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:34:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:34:19.924+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:34:20.266 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:34:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:34:19.924+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:34:20.266 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:34:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:34:19.924+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:34:20.267 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:34:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:34:19.925+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:34:20.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:34:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:34:20.265+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 11378 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:34:20.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:34:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:34:20.499+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:34:20.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:34:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:34:20.499+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:34:20.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:34:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:34:20.499+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:34:20.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:34:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:34:20.499+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:34:21.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:34:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:34:20.892+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:34:21.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:34:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:34:20.892+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:34:21.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:34:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:34:20.892+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:34:21.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:34:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:34:20.892+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:34:21.071 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:34:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:34:20.711+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:34:21.071 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:34:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:34:20.711+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:34:21.071 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:34:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:34:20.711+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:34:21.071 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:34:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:34:20.712+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:34:21.754 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:34:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:34:21.535+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:34:21.755 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:34:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:34:21.535+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:34:21.755 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:34:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:34:21.535+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:34:21.755 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:34:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:34:21.535+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:34:22.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:34:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:34:21.876+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:34:22.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:34:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:34:21.876+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:34:22.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:34:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:34:21.876+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:34:22.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:34:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:34:21.877+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:34:22.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:34:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:34:21.754+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:34:22.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:34:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:34:21.754+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:34:22.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:34:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:34:21.754+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:34:22.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:34:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:34:21.755+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:34:22.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:34:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:34:22.756+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:34:22.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:34:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:34:22.756+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:34:22.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:34:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:34:22.756+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:34:22.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:34:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:34:22.756+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:34:22.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:34:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:34:22.537+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:34:22.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:34:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:34:22.537+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:34:22.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:34:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:34:22.537+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:34:22.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:34:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:34:22.537+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:34:23.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:34:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:34:22.927+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:34:23.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:34:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:34:22.927+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:34:23.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:34:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:34:22.927+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:34:23.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:34:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:34:22.927+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:34:23.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:34:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:34:23.528+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:34:23.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:34:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:34:23.528+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:34:23.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:34:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:34:23.528+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:34:23.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:34:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:34:23.529+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:34:23.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:34:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:34:23.736+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:34:23.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:34:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:34:23.736+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:34:23.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:34:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:34:23.736+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:34:23.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:34:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:34:23.737+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:34:24.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:34:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:34:23.899+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:34:24.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:34:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:34:23.899+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:34:24.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:34:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:34:23.899+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:34:24.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:34:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:34:23.899+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:34:24.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:34:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:34:24.732+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:34:24.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:34:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:34:24.732+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:34:24.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:34:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:34:24.732+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:34:24.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:34:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:34:24.732+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:34:24.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:34:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:34:24.498+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:34:24.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:34:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:34:24.498+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:34:24.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:34:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:34:24.498+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:34:24.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:34:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:34:24.498+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:34:25.267 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:34:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:34:24.928+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:34:25.268 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:34:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:34:24.928+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:34:25.268 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:34:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:34:24.928+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:34:25.268 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:34:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:34:24.929+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:34:25.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:34:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:34:25.266+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 11473 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:34:25.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:34:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:34:25.483+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:34:25.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:34:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:34:25.483+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:34:25.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:34:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:34:25.483+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:34:25.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:34:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:34:25.484+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:34:26.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:34:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:34:25.923+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:34:26.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:34:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:34:25.923+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:34:26.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:34:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:34:25.923+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:34:26.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:34:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:34:25.923+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:34:26.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:34:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:34:25.706+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:34:26.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:34:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:34:25.706+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:34:26.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:34:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:34:25.706+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:34:26.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:34:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:34:25.707+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:34:26.708 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:34:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:34:26.436+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:34:26.708 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:34:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:34:26.436+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:34:26.708 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:34:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:34:26.436+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:34:26.708 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:34:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:34:26.437+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:34:27.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:34:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:34:26.707+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:34:27.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:34:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:34:26.707+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:34:27.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:34:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:34:26.707+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:34:27.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:34:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:34:26.708+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:34:27.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:34:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:34:26.875+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:34:27.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:34:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:34:26.875+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:34:27.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:34:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:34:26.875+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:34:27.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:34:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:34:26.876+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:34:27.740 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:34:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:34:27.439+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:34:27.741 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:34:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:34:27.440+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:34:27.741 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:34:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:34:27.440+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:34:27.742 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:34:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:34:27.440+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:34:28.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:34:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:34:27.855+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:34:28.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:34:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:34:27.855+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:34:28.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:34:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:34:27.855+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:34:28.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:34:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:34:27.855+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:34:28.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:34:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:34:27.740+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:34:28.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:34:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:34:27.740+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:34:28.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:34:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:34:27.740+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:34:28.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:34:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:34:27.741+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:34:28.759 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:34:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:34:28.400+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:34:28.760 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:34:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:34:28.400+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:34:28.760 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:34:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:34:28.400+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:34:28.760 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:34:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:34:28.400+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:34:29.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:34:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:34:28.881+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:34:29.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:34:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:34:28.881+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:34:29.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:34:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:34:28.881+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:34:29.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:34:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:34:28.882+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:34:29.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:34:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:34:28.758+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:34:29.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:34:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:34:28.758+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:34:29.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:34:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:34:28.758+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:34:29.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:34:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:34:28.759+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:34:29.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:34:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:34:29.361+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:34:29.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:34:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:34:29.361+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:34:29.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:34:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:34:29.361+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:34:29.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:34:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:34:29.362+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:34:30.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:34:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:34:29.776+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:34:30.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:34:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:34:29.776+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:34:30.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:34:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:34:29.776+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:34:30.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:34:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:34:29.777+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:34:30.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:34:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:34:29.858+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:34:30.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:34:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:34:29.858+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:34:30.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:34:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:34:29.858+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:34:30.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:34:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:34:29.858+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:34:30.577 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:34:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:34:30.268+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 11568 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:34:30.577 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:34:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:34:30.319+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:34:30.577 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:34:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:34:30.319+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:34:30.577 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:34:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:34:30.319+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:34:30.578 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:34:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:34:30.320+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:34:31.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:34:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:34:30.799+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:34:31.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:34:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:34:30.799+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:34:31.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:34:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:34:30.799+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:34:31.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:34:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:34:30.800+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:34:31.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:34:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:34:30.893+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:34:31.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:34:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:34:30.893+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:34:31.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:34:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:34:30.893+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:34:31.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:34:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:34:30.894+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:34:31.678 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:34:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:34:31.341+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:34:31.679 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:34:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:34:31.341+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:34:31.679 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:34:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:34:31.341+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:34:31.679 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:34:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:34:31.341+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:34:32.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:34:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:34:31.842+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:34:32.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:34:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:34:31.842+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:34:32.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:34:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:34:31.842+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:34:32.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:34:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:34:31.843+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:34:32.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:34:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:34:31.881+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:34:32.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:34:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:34:31.881+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:34:32.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:34:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:34:31.881+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:34:32.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:34:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:34:31.881+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:34:32.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:34:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:34:32.380+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:34:32.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:34:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:34:32.380+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:34:32.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:34:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:34:32.380+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:34:32.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:34:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:34:32.381+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:34:33.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:34:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:34:32.899+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:34:33.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:34:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:34:32.899+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:34:33.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:34:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:34:32.899+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:34:33.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:34:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:34:32.899+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:34:33.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:34:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:34:32.878+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:34:33.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:34:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:34:32.878+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:34:33.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:34:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:34:32.878+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:34:33.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:34:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:34:32.879+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:34:33.754 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:34:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:34:33.390+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:34:33.755 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:34:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:34:33.390+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:34:33.755 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:34:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:34:33.390+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:34:33.755 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:34:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:34:33.391+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:34:34.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:34:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:34:33.879+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:34:34.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:34:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:34:33.879+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:34:34.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:34:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:34:33.879+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:34:34.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:34:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:34:33.880+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:34:34.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:34:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:34:33.922+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:34:34.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:34:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:34:33.922+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:34:34.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:34:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:34:33.922+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:34:34.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:34:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:34:33.922+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:34:34.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:34:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:34:34.360+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:34:34.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:34:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:34:34.360+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:34:34.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:34:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:34:34.360+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:34:34.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:34:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:34:34.361+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:34:35.270 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:34:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:34:34.873+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:34:35.271 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:34:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:34:34.873+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:34:35.271 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:34:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:34:34.873+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:34:35.271 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:34:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:34:34.873+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:34:35.271 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:34:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:34:34.850+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:34:35.271 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:34:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:34:34.850+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:34:35.271 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:34:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:34:34.850+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:34:35.272 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:34:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:34:34.850+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:34:35.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:34:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:34:35.269+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 11662 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:34:35.587 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:34:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:34:35.392+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:34:35.587 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:34:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:34:35.392+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:34:35.587 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:34:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:34:35.392+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:34:35.587 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:34:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:34:35.393+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:34:36.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:34:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:34:35.887+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:34:36.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:34:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:34:35.887+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:34:36.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:34:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:34:35.887+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:34:36.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:34:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:34:35.888+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:34:36.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:34:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:34:35.840+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:34:36.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:34:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:34:35.840+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:34:36.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:34:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:34:35.840+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:34:36.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:34:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:34:35.840+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:34:36.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:34:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:34:36.386+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:34:36.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:34:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:34:36.386+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:34:36.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:34:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:34:36.386+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:34:36.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:34:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:34:36.387+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:34:37.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:34:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:34:36.874+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:34:37.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:34:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:34:36.874+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:34:37.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:34:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:34:36.874+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:34:37.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:34:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:34:36.875+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:34:37.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:34:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:34:36.818+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:34:37.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:34:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:34:36.818+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:34:37.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:34:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:34:36.818+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:34:37.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:34:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:34:36.819+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:34:37.791 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:34:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:34:37.355+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:34:37.791 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:34:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:34:37.355+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:34:37.791 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:34:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:34:37.355+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:34:37.791 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:34:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:34:37.356+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:34:38.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:34:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:34:37.848+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:34:38.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:34:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:34:37.848+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:34:38.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:34:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:34:37.848+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:34:38.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:34:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:34:37.849+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:34:38.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:34:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:34:37.790+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:34:38.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:34:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:34:37.790+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:34:38.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:34:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:34:37.790+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:34:38.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:34:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:34:37.790+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:34:38.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:34:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:34:38.348+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:34:38.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:34:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:34:38.348+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:34:38.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:34:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:34:38.348+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:34:38.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:34:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:34:38.349+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:34:39.115 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:34:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:34:38.820+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:34:39.115 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:34:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:34:38.820+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:34:39.116 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:34:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:34:38.820+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:34:39.116 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:34:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:34:38.821+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:34:39.116 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:34:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:34:38.823+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:34:39.116 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:34:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:34:38.823+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:34:39.116 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:34:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:34:38.823+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:34:39.116 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:34:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:34:38.823+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:34:39.365 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:34:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:34:39.364+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:34:39.366 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:34:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:34:39.364+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:34:39.366 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:34:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:34:39.364+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:34:39.804 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:34:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:34:39.365+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:34:40.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:34:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:34:39.848+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:34:40.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:34:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:34:39.848+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:34:40.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:34:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:34:39.848+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:34:40.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:34:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:34:39.849+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:34:40.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:34:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:34:39.803+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:34:40.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:34:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:34:39.803+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:34:40.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:34:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:34:39.803+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:34:40.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:34:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:34:39.804+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:34:40.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:34:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:34:40.271+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 11756 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:34:40.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:34:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:34:40.339+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:34:40.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:34:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:34:40.339+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:34:40.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:34:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:34:40.339+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:34:40.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:34:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:34:40.340+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:34:41.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:34:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:34:40.842+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:34:41.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:34:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:34:40.842+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:34:41.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:34:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:34:40.842+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:34:41.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:34:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:34:40.842+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:34:41.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:34:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:34:40.779+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:34:41.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:34:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:34:40.779+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:34:41.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:34:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:34:40.779+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:34:41.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:34:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:34:40.779+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:34:41.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:34:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:34:41.296+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:34:41.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:34:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:34:41.296+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:34:41.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:34:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:34:41.296+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:34:41.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:34:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:34:41.297+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:34:42.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:34:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:34:41.811+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:34:42.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:34:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:34:41.811+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:34:42.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:34:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:34:41.811+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:34:42.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:34:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:34:41.812+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:34:42.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:34:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:34:41.823+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:34:42.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:34:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:34:41.823+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:34:42.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:34:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:34:41.823+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:34:42.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:34:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:34:41.824+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:34:42.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:34:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:34:42.301+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:34:42.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:34:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:34:42.301+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:34:42.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:34:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:34:42.301+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:34:42.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:34:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:34:42.302+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:34:43.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:34:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:34:42.776+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:34:43.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:34:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:34:42.776+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:34:43.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:34:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:34:42.776+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:34:43.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:34:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:34:42.777+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:34:43.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:34:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:34:42.874+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:34:43.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:34:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:34:42.874+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:34:43.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:34:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:34:42.874+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:34:43.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:34:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:34:42.874+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:34:43.763 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:34:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:34:43.320+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:34:43.763 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:34:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:34:43.320+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:34:43.764 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:34:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:34:43.320+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:34:43.764 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:34:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:34:43.321+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:34:44.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:34:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:34:43.762+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:34:44.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:34:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:34:43.762+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:34:44.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:34:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:34:43.762+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:34:44.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:34:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:34:43.764+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:34:44.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:34:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:34:43.888+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:34:44.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:34:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:34:43.888+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:34:44.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:34:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:34:43.888+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:34:44.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:34:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:34:43.888+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:34:44.743 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:34:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:34:44.345+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:34:44.743 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:34:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:34:44.345+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:34:44.744 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:34:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:34:44.345+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:34:44.744 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:34:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:34:44.346+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:34:45.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:34:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:34:44.742+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:34:45.071 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:34:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:34:44.742+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:34:45.071 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:34:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:34:44.742+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:34:45.071 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:34:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:34:44.743+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:34:45.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:34:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:34:44.851+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:34:45.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:34:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:34:44.851+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:34:45.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:34:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:34:44.851+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:34:45.072 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:34:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:34:44.851+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:34:45.482 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:34:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:34:45.272+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 11850 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:34:45.482 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:34:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:34:45.327+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:34:45.483 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:34:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:34:45.327+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:34:45.483 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:34:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:34:45.327+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:34:45.483 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:34:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:34:45.328+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:34:46.043 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:34:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:34:45.862+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:34:46.043 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:34:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:34:45.862+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:34:46.043 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:34:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:34:45.862+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:34:46.043 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:34:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:34:45.862+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:34:46.043 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:34:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:34:45.773+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:34:46.044 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:34:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:34:45.773+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:34:46.044 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:34:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:34:45.773+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:34:46.044 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:34:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:34:45.773+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:34:46.814 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:34:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:34:46.321+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:34:46.815 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:34:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:34:46.321+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:34:46.815 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:34:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:34:46.321+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:34:46.815 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:34:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:34:46.322+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:34:47.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:34:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:34:46.814+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:34:47.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:34:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:34:46.814+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:34:47.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:34:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:34:46.814+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:34:47.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:34:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:34:46.814+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:34:47.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:34:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:34:46.838+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:34:47.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:34:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:34:46.838+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:34:47.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:34:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:34:46.838+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:34:47.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:34:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:34:46.838+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:34:47.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:34:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:34:47.312+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:34:47.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:34:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:34:47.312+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:34:47.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:34:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:34:47.312+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:34:47.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:34:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:34:47.312+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:34:48.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:34:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:34:47.866+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:34:48.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:34:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:34:47.866+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:34:48.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:34:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:34:47.866+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:34:48.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:34:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:34:47.867+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:34:48.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:34:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:34:47.819+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:34:48.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:34:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:34:47.819+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:34:48.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:34:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:34:47.819+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:34:48.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:34:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:34:47.820+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:34:48.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:34:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:34:48.336+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:34:48.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:34:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:34:48.336+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:34:48.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:34:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:34:48.336+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:34:48.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:34:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:34:48.337+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:34:49.116 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:34:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:34:48.837+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:34:49.116 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:34:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:34:48.837+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:34:49.117 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:34:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:34:48.837+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:34:49.117 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:34:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:34:48.837+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:34:49.117 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:34:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:34:48.909+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:34:49.117 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:34:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:34:48.909+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:34:49.117 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:34:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:34:48.909+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:34:49.117 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:34:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:34:48.910+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:34:49.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:34:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:34:49.354+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:34:49.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:34:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:34:49.354+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:34:49.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:34:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:34:49.354+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:34:49.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:34:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:34:49.355+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:34:50.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:34:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:34:49.791+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:34:50.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:34:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:34:49.791+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:34:50.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:34:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:34:49.791+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:34:50.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:34:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:34:49.791+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:34:50.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:34:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:34:49.914+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:34:50.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:34:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:34:49.914+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:34:50.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:34:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:34:49.914+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:34:50.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:34:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:34:49.915+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:34:50.399 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:34:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:34:50.272+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 11945 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:34:50.399 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:34:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:34:50.391+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:34:50.399 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:34:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:34:50.391+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:34:50.400 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:34:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:34:50.391+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:34:50.400 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:34:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:34:50.392+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:34:51.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:34:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:34:50.765+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:34:51.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:34:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:34:50.765+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:34:51.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:34:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:34:50.765+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:34:51.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:34:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:34:50.766+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:34:51.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:34:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:34:50.912+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:34:51.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:34:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:34:50.912+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:34:51.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:34:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:34:50.912+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:34:51.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:34:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:34:50.912+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:34:51.680 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:34:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:34:51.362+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:34:51.680 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:34:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:34:51.362+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:34:51.680 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:34:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:34:51.362+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:34:51.680 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:34:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:34:51.362+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:34:51.938 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:34:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:34:51.735+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:34:51.938 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:34:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:34:51.735+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:34:51.938 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:34:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:34:51.735+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:34:51.938 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:34:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:34:51.736+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:34:52.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:34:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:34:51.936+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:34:52.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:34:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:34:51.936+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:34:52.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:34:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:34:51.936+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:34:52.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:34:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:34:51.937+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:34:52.704 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:34:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:34:52.355+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:34:52.704 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:34:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:34:52.355+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:34:52.704 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:34:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:34:52.355+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:34:52.705 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:34:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:34:52.355+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:34:52.959 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:34:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:34:52.702+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:34:52.960 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:34:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:34:52.702+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:34:52.960 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:34:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:34:52.702+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:34:52.960 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:34:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:34:52.703+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:34:53.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:34:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:34:52.957+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:34:53.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:34:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:34:52.957+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:34:53.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:34:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:34:52.957+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:34:53.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:34:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:34:52.958+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:34:53.657 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:34:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:34:53.371+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:34:53.657 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:34:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:34:53.371+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:34:53.657 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:34:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:34:53.371+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:34:53.657 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:34:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:34:53.372+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:34:54.003 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:34:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:34:53.655+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:34:54.004 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:34:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:34:53.655+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:34:54.004 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:34:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:34:53.655+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:34:54.004 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:34:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:34:53.656+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:34:54.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:34:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:34:54.002+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:34:54.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:34:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:34:54.002+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:34:54.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:34:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:34:54.002+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:34:54.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:34:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:34:54.002+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:34:54.631 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:34:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:34:54.343+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:34:54.632 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:34:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:34:54.343+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:34:54.632 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:34:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:34:54.343+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:34:54.632 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:34:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:34:54.343+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:34:55.030 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:34:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:34:54.629+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:34:55.030 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:34:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:34:54.629+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:34:55.030 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:34:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:34:54.629+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:34:55.030 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:34:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:34:54.630+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:34:55.287 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:34:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:34:55.274+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 12039 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:34:55.287 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:34:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:34:55.028+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:34:55.287 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:34:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:34:55.028+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:34:55.287 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:34:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:34:55.028+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:34:55.288 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:34:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:34:55.029+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:34:55.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:34:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:34:55.308+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:34:55.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:34:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:34:55.308+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:34:55.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:34:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:34:55.308+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:34:55.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:34:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:34:55.309+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:34:55.826 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:34:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:34:55.625+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:34:55.826 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:34:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:34:55.625+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:34:55.827 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:34:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:34:55.625+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:34:55.827 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:34:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:34:55.625+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:34:56.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:34:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:34:56.017+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:34:56.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:34:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:34:56.017+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:34:56.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:34:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:34:56.017+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:34:56.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:34:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:34:56.017+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:34:56.573 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:34:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:34:56.270+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:34:56.574 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:34:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:34:56.270+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:34:56.574 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:34:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:34:56.270+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:34:56.574 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:34:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:34:56.271+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:34:56.857 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:34:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:34:56.586+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:34:56.858 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:34:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:34:56.586+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:34:56.858 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:34:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:34:56.586+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:34:56.858 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:34:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:34:56.587+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:34:57.251 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:34:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:34:57.032+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:34:57.251 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:34:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:34:57.032+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:34:57.251 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:34:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:34:57.032+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:34:57.251 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:34:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:34:57.033+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:34:57.567 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:34:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:34:57.249+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:34:57.567 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:34:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:34:57.249+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:34:57.567 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:34:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:34:57.249+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:34:57.567 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:34:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:34:57.250+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:34:57.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:34:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:34:57.565+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:34:57.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:34:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:34:57.565+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:34:57.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:34:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:34:57.565+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:34:57.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:34:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:34:57.566+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:34:58.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:34:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:34:58.020+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:34:58.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:34:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:34:58.020+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:34:58.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:34:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:34:58.020+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:34:58.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:34:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:34:58.020+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:34:58.321 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:34:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:34:58.233+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:34:58.321 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:34:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:34:58.233+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:34:58.321 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:34:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:34:58.233+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:34:58.322 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:34:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:34:58.234+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:34:58.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:34:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:34:58.556+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:34:58.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:34:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:34:58.556+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:34:58.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:34:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:34:58.556+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:34:58.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:34:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:34:58.557+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:34:59.281 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:34:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:34:59.007+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:34:59.281 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:34:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:34:59.007+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:34:59.282 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:34:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:34:59.007+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:34:59.282 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:34:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:34:59.008+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:34:59.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:34:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:34:59.280+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:34:59.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:34:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:34:59.280+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:34:59.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:34:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:34:59.280+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:34:59.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:34:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:34:59.280+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:35:00.019 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:34:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:34:59.577+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:35:00.020 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:34:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:34:59.577+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:35:00.020 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:34:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:34:59.577+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:35:00.020 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:34:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:34:59.578+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:35:00.277 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:35:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:35:00.017+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:35:00.277 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:35:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:35:00.017+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:35:00.277 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:35:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:35:00.017+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:35:00.277 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:35:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:35:00.019+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:35:00.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:35:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:35:00.276+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 12133 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:35:00.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:35:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:35:00.291+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:35:00.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:35:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:35:00.291+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:35:00.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:35:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:35:00.291+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:35:00.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:35:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:35:00.292+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:35:01.012 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:35:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:35:00.588+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:35:01.013 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:35:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:35:00.588+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:35:01.013 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:35:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:35:00.588+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:35:01.013 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:35:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:35:00.589+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:35:01.283 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:35:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:35:01.011+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:35:01.283 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:35:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:35:01.011+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:35:01.283 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:35:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:35:01.011+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:35:01.283 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:35:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:35:01.012+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:35:01.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:35:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:35:01.282+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:35:01.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:35:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:35:01.282+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:35:01.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:35:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:35:01.282+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:35:01.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:35:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:35:01.283+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:35:01.969 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:35:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:35:01.618+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:35:01.969 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:35:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:35:01.618+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:35:01.969 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:35:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:35:01.618+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:35:01.969 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:35:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:35:01.619+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:35:02.294 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:35:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:35:01.968+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:35:02.295 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:35:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:35:01.968+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:35:02.295 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:35:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:35:01.968+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:35:02.295 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:35:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:35:01.968+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:35:02.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:35:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:35:02.293+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:35:02.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:35:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:35:02.293+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:35:02.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:35:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:35:02.293+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:35:02.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:35:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:35:02.294+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:35:03.015 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:35:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:35:02.659+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:35:03.015 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:35:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:35:02.659+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:35:03.015 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:35:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:35:02.659+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:35:03.015 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:35:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:35:02.659+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:35:03.297 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:35:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:35:03.014+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:35:03.297 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:35:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:35:03.014+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:35:03.298 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:35:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:35:03.014+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:35:03.298 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:35:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:35:03.014+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:35:03.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:35:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:35:03.296+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:35:03.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:35:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:35:03.296+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:35:03.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:35:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:35:03.296+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:35:03.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:35:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:35:03.297+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:35:04.035 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:35:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:35:03.640+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:35:04.035 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:35:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:35:03.640+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:35:04.035 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:35:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:35:03.640+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:35:04.035 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:35:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:35:03.641+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:35:04.319 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:35:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:35:04.273+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:35:04.319 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:35:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:35:04.273+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:35:04.319 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:35:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:35:04.273+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:35:04.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:35:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:35:04.274+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:35:04.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:35:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:35:04.034+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:35:04.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:35:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:35:04.034+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:35:04.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:35:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:35:04.034+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:35:04.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:35:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:35:04.034+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:35:04.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:35:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:35:04.592+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:35:04.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:35:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:35:04.592+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:35:04.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:35:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:35:04.592+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:35:04.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:35:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:35:04.593+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:35:05.247 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:35:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:35:04.986+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:35:05.247 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:35:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:35:04.986+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:35:05.248 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:35:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:35:04.986+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:35:05.248 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:35:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:35:04.986+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:35:05.545 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:35:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:35:05.277+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 12228 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:35:05.545 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:35:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:35:05.246+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:35:05.546 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:35:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:35:05.246+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:35:05.546 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:35:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:35:05.246+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:35:05.546 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:35:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:35:05.247+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:35:05.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:35:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:35:05.544+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:35:05.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:35:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:35:05.544+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:35:05.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:35:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:35:05.544+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:35:05.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:35:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:35:05.546+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:35:06.257 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:35:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:35:05.985+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:35:06.257 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:35:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:35:05.985+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:35:06.257 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:35:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:35:05.985+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:35:06.257 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:35:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:35:05.985+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:35:06.562 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:35:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:35:06.256+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:35:06.563 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:35:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:35:06.256+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:35:06.563 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:35:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:35:06.256+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:35:06.563 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:35:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:35:06.256+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:35:06.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:35:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:35:06.562+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:35:06.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:35:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:35:06.562+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:35:06.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:35:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:35:06.562+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:35:06.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:35:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:35:06.562+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:35:07.212 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:35:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:35:06.937+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:35:07.212 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:35:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:35:06.937+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:35:07.212 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:35:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:35:06.937+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:35:07.212 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:35:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:35:06.937+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:35:07.536 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:35:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:35:07.211+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:35:07.536 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:35:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:35:07.211+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:35:07.536 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:35:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:35:07.211+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:35:07.536 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:35:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:35:07.211+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:35:07.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:35:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:35:07.535+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:35:07.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:35:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:35:07.535+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:35:07.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:35:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:35:07.535+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:35:07.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:35:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:35:07.536+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:35:08.319 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:35:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:35:08.208+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:35:08.319 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:35:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:35:08.208+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:35:08.319 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:35:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:35:08.208+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:35:08.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:35:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:35:08.208+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:35:08.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:35:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:35:07.987+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:35:08.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:35:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:35:07.987+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:35:08.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:35:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:35:07.987+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:35:08.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:35:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:35:07.988+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:35:08.947 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:35:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:35:08.578+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:35:08.947 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:35:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:35:08.578+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:35:08.947 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:35:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:35:08.578+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:35:08.947 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:35:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:35:08.579+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:35:09.202 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:35:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:35:08.946+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:35:09.202 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:35:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:35:08.946+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:35:09.203 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:35:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:35:08.946+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:35:09.203 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:35:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:35:08.946+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:35:09.555 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:35:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:35:09.201+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:35:09.555 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:35:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:35:09.201+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:35:09.555 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:35:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:35:09.201+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:35:09.555 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:35:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:35:09.202+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:35:09.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:35:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:35:09.554+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:35:09.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:35:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:35:09.554+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:35:09.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:35:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:35:09.554+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:35:09.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:35:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:35:09.555+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:35:10.213 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:35:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:35:09.950+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:35:10.213 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:35:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:35:09.950+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:35:10.214 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:35:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:35:09.950+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:35:10.214 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:35:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:35:09.950+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:35:10.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:35:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:35:10.278+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 12323 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:35:10.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:35:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:35:10.212+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:35:10.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:35:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:35:10.212+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:35:10.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:35:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:35:10.212+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:35:10.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:35:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:35:10.213+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:35:10.906 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:35:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:35:10.599+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:35:10.906 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:35:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:35:10.599+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:35:10.907 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:35:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:35:10.599+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:35:10.907 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:35:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:35:10.600+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:35:11.211 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:35:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:35:10.906+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:35:11.211 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:35:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:35:10.906+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:35:11.211 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:35:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:35:10.906+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:35:11.211 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:35:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:35:10.906+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:35:11.568 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:35:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:35:11.210+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:35:11.568 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:35:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:35:11.210+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:35:11.568 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:35:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:35:11.210+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:35:11.568 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:35:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:35:11.211+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:35:11.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:35:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:35:11.567+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:35:11.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:35:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:35:11.567+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:35:11.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:35:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:35:11.567+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:35:11.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:35:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:35:11.568+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:35:12.217 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:35:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:35:11.861+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:35:12.218 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:35:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:35:11.861+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:35:12.218 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:35:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:35:11.861+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:35:12.218 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:35:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:35:11.861+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:35:12.533 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:35:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:35:12.216+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:35:12.533 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:35:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:35:12.216+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:35:12.534 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:35:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:35:12.216+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:35:12.534 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:35:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:35:12.217+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:35:12.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:35:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:35:12.532+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:35:12.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:35:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:35:12.532+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:35:12.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:35:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:35:12.532+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:35:12.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:35:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:35:12.533+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:35:13.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:35:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:35:12.861+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:35:13.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:35:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:35:12.861+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:35:13.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:35:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:35:12.861+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:35:13.222 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:35:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:35:12.861+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:35:13.566 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:35:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:35:13.220+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:35:13.566 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:35:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:35:13.220+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:35:13.566 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:35:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:35:13.220+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:35:13.566 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:35:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:35:13.221+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:35:13.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:35:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:35:13.565+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:35:13.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:35:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:35:13.565+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:35:13.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:35:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:35:13.565+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:35:13.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:35:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:35:13.566+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:35:14.200 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:35:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:35:13.884+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:35:14.200 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:35:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:35:13.884+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:35:14.200 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:35:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:35:13.884+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:35:14.200 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:35:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:35:13.884+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:35:14.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:35:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:35:14.199+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:35:14.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:35:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:35:14.199+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:35:14.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:35:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:35:14.199+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:35:14.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:35:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:35:14.200+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:35:14.844 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:35:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:35:14.581+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:35:14.844 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:35:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:35:14.581+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:35:14.844 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:35:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:35:14.581+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:35:14.845 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:35:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:35:14.582+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:35:15.201 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:35:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:35:14.843+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:35:15.201 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:35:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:35:14.843+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:35:15.201 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:35:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:35:14.843+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:35:15.201 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:35:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:35:14.844+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:35:15.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:35:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:35:15.280+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 12417 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:35:15.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:35:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:35:15.200+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:35:15.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:35:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:35:15.200+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:35:15.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:35:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:35:15.200+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:35:15.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:35:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:35:15.201+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:35:16.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:35:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:35:15.866+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:35:16.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:35:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:35:15.866+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:35:16.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:35:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:35:15.866+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:35:16.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:35:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:35:15.866+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:35:16.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:35:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:35:15.626+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:35:16.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:35:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:35:15.626+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:35:16.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:35:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:35:15.626+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:35:16.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:35:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:35:15.627+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:35:16.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:35:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:35:16.246+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:35:16.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:35:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:35:16.246+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:35:16.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:35:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:35:16.246+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:35:16.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:35:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:35:16.247+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:35:17.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:35:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:35:16.598+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:35:17.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:35:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:35:16.598+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:35:17.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:35:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:35:16.598+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:35:17.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:35:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:35:16.599+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:35:17.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:35:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:35:16.825+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:35:17.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:35:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:35:16.825+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:35:17.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:35:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:35:16.825+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:35:17.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:35:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:35:16.825+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:35:17.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:35:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:35:17.255+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:35:17.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:35:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:35:17.255+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:35:17.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:35:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:35:17.255+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:35:17.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:35:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:35:17.255+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:35:18.072 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:35:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:35:17.614+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:35:18.072 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:35:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:35:17.614+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:35:18.072 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:35:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:35:17.614+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:35:18.072 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:35:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:35:17.615+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:35:18.073 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:35:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:35:17.837+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:35:18.073 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:35:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:35:17.837+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:35:18.073 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:35:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:35:17.837+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:35:18.073 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:35:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:35:17.837+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:35:18.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:35:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:35:18.254+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:35:18.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:35:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:35:18.254+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:35:18.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:35:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:35:18.254+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:35:18.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:35:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:35:18.254+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:35:19.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:35:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:35:18.655+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:35:19.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:35:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:35:18.655+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:35:19.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:35:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:35:18.655+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:35:19.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:35:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:35:18.656+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:35:19.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:35:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:35:18.795+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:35:19.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:35:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:35:18.795+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:35:19.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:35:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:35:18.795+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:35:19.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:35:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:35:18.795+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:35:19.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:35:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:35:19.266+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:35:19.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:35:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:35:19.266+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:35:19.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:35:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:35:19.266+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:35:19.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:35:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:35:19.266+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:35:20.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:35:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:35:19.757+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:35:20.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:35:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:35:19.757+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:35:20.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:35:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:35:19.757+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:35:20.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:35:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:35:19.757+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:35:20.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:35:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:35:19.698+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:35:20.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:35:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:35:19.698+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:35:20.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:35:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:35:19.698+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:35:20.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:35:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:35:19.699+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:35:20.407 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:35:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:35:20.281+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 12511 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:35:20.407 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:35:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:35:20.268+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:35:20.407 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:35:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:35:20.268+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:35:20.407 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:35:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:35:20.268+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:35:20.407 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:35:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:35:20.269+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:35:21.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:35:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:35:20.694+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:35:21.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:35:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:35:20.694+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:35:21.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:35:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:35:20.694+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:35:21.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:35:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:35:20.695+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:35:21.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:35:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:35:20.726+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:35:21.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:35:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:35:20.726+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:35:21.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:35:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:35:20.726+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:35:21.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:35:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:35:20.726+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:35:21.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:35:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:35:21.281+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:35:21.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:35:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:35:21.281+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:35:21.571 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:35:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:35:21.281+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:35:21.571 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:35:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:35:21.282+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:35:22.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:35:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:35:21.698+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:35:22.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:35:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:35:21.698+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:35:22.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:35:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:35:21.698+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:35:22.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:35:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:35:21.699+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:35:22.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:35:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:35:21.737+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:35:22.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:35:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:35:21.737+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:35:22.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:35:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:35:21.737+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:35:22.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:35:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:35:21.737+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:35:22.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:35:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:35:22.293+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:35:22.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:35:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:35:22.293+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:35:22.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:35:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:35:22.293+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:35:22.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:35:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:35:22.294+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:35:23.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:35:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:35:22.677+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:35:23.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:35:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:35:22.677+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:35:23.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:35:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:35:22.677+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:35:23.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:35:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:35:22.678+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:35:23.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:35:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:35:22.767+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:35:23.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:35:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:35:22.767+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:35:23.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:35:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:35:22.767+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:35:23.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:35:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:35:22.768+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:35:23.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:35:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:35:23.297+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:35:23.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:35:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:35:23.297+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:35:23.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:35:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:35:23.297+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:35:23.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:35:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:35:23.298+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:35:24.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:35:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:35:23.699+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:35:24.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:35:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:35:23.699+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:35:24.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:35:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:35:23.699+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:35:24.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:35:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:35:23.700+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:35:24.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:35:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:35:23.737+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:35:24.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:35:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:35:23.737+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:35:24.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:35:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:35:23.737+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:35:24.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:35:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:35:23.737+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:35:24.659 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:35:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:35:24.336+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:35:24.659 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:35:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:35:24.336+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:35:24.659 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:35:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:35:24.336+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:35:24.659 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:35:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:35:24.336+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:35:25.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:35:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:35:24.658+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:35:25.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:35:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:35:24.658+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:35:25.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:35:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:35:24.658+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:35:25.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:35:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:35:24.659+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:35:25.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:35:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:35:24.707+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:35:25.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:35:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:35:24.707+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:35:25.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:35:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:35:24.707+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:35:25.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:35:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:35:24.708+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:35:25.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:35:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:35:25.283+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 12606 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:35:25.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:35:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:35:25.292+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:35:25.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:35:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:35:25.292+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:35:25.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:35:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:35:25.292+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:35:25.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:35:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:35:25.293+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:35:25.979 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:35:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:35:25.637+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:35:25.979 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:35:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:35:25.637+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:35:25.979 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:35:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:35:25.637+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:35:25.979 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:35:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:35:25.637+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:35:25.980 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:35:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:35:25.663+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:35:25.980 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:35:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:35:25.663+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:35:25.980 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:35:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:35:25.663+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:35:25.980 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:35:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:35:25.663+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:35:26.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:35:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:35:26.243+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:35:26.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:35:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:35:26.243+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:35:26.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:35:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:35:26.243+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:35:26.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:35:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:35:26.244+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:35:27.075 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:35:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:35:26.681+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:35:27.076 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:35:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:35:26.681+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:35:27.076 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:35:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:35:26.681+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:35:27.076 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:35:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:35:26.682+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:35:27.077 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:35:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:35:26.618+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:35:27.077 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:35:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:35:26.618+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:35:27.077 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:35:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:35:26.618+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:35:27.078 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:35:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:35:26.618+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:35:27.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:35:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:35:27.255+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:35:27.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:35:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:35:27.255+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:35:27.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:35:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:35:27.255+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:35:27.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:35:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:35:27.256+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:35:28.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:35:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:35:27.661+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:35:28.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:35:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:35:27.661+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:35:28.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:35:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:35:27.661+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:35:28.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:35:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:35:27.662+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:35:28.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:35:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:35:27.672+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:35:28.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:35:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:35:27.672+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:35:28.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:35:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:35:27.672+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:35:28.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:35:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:35:27.673+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:35:28.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:35:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:35:28.241+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:35:28.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:35:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:35:28.241+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:35:28.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:35:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:35:28.241+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:35:28.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:35:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:35:28.242+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:35:29.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:35:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:35:28.614+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:35:29.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:35:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:35:28.614+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:35:29.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:35:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:35:28.614+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:35:29.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:35:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:35:28.614+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:35:29.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:35:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:35:28.691+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:35:29.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:35:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:35:28.691+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:35:29.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:35:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:35:28.691+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:35:29.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:35:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:35:28.692+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:35:29.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:35:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:35:29.212+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:35:29.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:35:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:35:29.212+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:35:29.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:35:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:35:29.212+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:35:29.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:35:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:35:29.212+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:35:30.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:35:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:35:29.717+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:35:30.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:35:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:35:29.717+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:35:30.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:35:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:35:29.717+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:35:30.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:35:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:35:29.718+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:35:30.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:35:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:35:29.654+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:35:30.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:35:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:35:29.654+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:35:30.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:35:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:35:29.654+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:35:30.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:35:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:35:29.654+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:35:30.405 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:35:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:35:30.284+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 12701 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:35:30.405 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:35:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:35:30.206+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:35:30.406 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:35:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:35:30.206+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:35:30.406 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:35:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:35:30.206+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:35:30.406 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:35:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:35:30.206+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:35:31.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:35:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:35:30.724+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:35:31.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:35:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:35:30.724+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:35:31.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:35:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:35:30.724+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:35:31.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:35:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:35:30.724+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:35:31.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:35:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:35:30.674+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:35:31.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:35:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:35:30.674+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:35:31.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:35:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:35:30.674+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:35:31.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:35:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:35:30.675+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:35:31.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:35:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:35:31.231+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:35:31.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:35:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:35:31.231+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:35:31.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:35:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:35:31.231+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:35:31.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:35:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:35:31.231+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:35:32.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:35:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:35:31.649+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:35:32.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:35:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:35:31.649+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:35:32.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:35:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:35:31.649+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:35:32.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:35:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:35:31.649+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:35:32.071 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:35:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:35:31.764+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:35:32.071 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:35:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:35:31.764+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:35:32.071 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:35:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:35:31.764+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:35:32.072 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:35:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:35:31.765+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:35:32.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:35:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:35:32.270+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:35:32.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:35:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:35:32.270+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:35:32.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:35:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:35:32.270+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:35:32.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:35:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:35:32.271+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:35:33.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:35:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:35:32.762+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:35:33.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:35:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:35:32.762+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:35:33.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:35:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:35:32.762+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:35:33.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:35:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:35:32.763+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:35:33.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:35:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:35:32.680+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:35:33.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:35:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:35:32.680+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:35:33.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:35:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:35:32.680+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:35:33.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:35:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:35:32.680+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:35:33.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:35:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:35:33.270+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:35:33.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:35:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:35:33.270+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:35:33.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:35:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:35:33.270+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:35:33.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:35:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:35:33.271+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:35:34.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:35:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:35:33.806+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:35:34.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:35:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:35:33.806+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:35:34.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:35:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:35:33.806+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:35:34.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:35:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:35:33.807+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:35:34.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:35:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:35:33.656+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:35:34.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:35:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:35:33.656+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:35:34.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:35:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:35:33.656+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:35:34.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:35:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:35:33.656+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:35:34.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:35:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:35:34.228+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:35:34.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:35:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:35:34.228+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:35:34.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:35:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:35:34.228+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:35:34.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:35:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:35:34.229+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:35:35.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:35:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:35:34.769+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:35:35.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:35:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:35:34.769+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:35:35.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:35:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:35:34.769+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:35:35.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:35:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:35:34.769+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:35:35.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:35:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:35:34.639+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:35:35.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:35:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:35:34.639+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:35:35.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:35:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:35:34.639+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:35:35.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:35:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:35:34.639+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:35:35.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:35:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:35:35.285+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 12795 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:35:35.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:35:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:35:35.191+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:35:35.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:35:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:35:35.191+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:35:35.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:35:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:35:35.191+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:35:35.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:35:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:35:35.191+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:35:36.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:35:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:35:35.740+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:35:36.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:35:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:35:35.740+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:35:36.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:35:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:35:35.740+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:35:36.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:35:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:35:35.741+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:35:36.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:35:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:35:35.633+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:35:36.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:35:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:35:35.633+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:35:36.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:35:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:35:35.633+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:35:36.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:35:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:35:35.633+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:35:36.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:35:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:35:36.213+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:35:36.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:35:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:35:36.213+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:35:36.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:35:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:35:36.213+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:35:36.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:35:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:35:36.214+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:35:37.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:35:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:35:36.723+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:35:37.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:35:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:35:36.723+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:35:37.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:35:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:35:36.723+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:35:37.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:35:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:35:36.724+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:35:37.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:35:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:35:36.652+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:35:37.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:35:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:35:36.652+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:35:37.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:35:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:35:36.652+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:35:37.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:35:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:35:36.652+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:35:37.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:35:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:35:37.248+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:35:37.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:35:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:35:37.248+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:35:37.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:35:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:35:37.248+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:35:37.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:35:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:35:37.249+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:35:38.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:35:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:35:37.632+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:35:38.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:35:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:35:37.632+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:35:38.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:35:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:35:37.632+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:35:38.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:35:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:35:37.633+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:35:38.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:35:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:35:37.700+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:35:38.071 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:35:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:35:37.700+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:35:38.071 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:35:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:35:37.700+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:35:38.071 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:35:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:35:37.701+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:35:38.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:35:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:35:38.205+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:35:38.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:35:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:35:38.205+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:35:38.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:35:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:35:38.205+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:35:38.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:35:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:35:38.206+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:35:39.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:35:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:35:38.692+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:35:39.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:35:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:35:38.692+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:35:39.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:35:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:35:38.692+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:35:39.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:35:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:35:38.693+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:35:39.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:35:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:35:38.637+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:35:39.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:35:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:35:38.637+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:35:39.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:35:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:35:38.637+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:35:39.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:35:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:35:38.638+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:35:39.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:35:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:35:39.200+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:35:39.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:35:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:35:39.200+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:35:39.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:35:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:35:39.200+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:35:39.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:35:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:35:39.201+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:35:40.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:35:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:35:39.730+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:35:40.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:35:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:35:39.730+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:35:40.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:35:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:35:39.730+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:35:40.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:35:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:35:39.730+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:35:40.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:35:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:35:39.651+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:35:40.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:35:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:35:39.651+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:35:40.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:35:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:35:39.651+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:35:40.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:35:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:35:39.652+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:35:40.407 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:35:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:35:40.286+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 12888 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:35:40.407 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:35:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:35:40.152+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:35:40.408 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:35:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:35:40.152+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:35:40.408 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:35:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:35:40.152+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:35:40.408 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:35:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:35:40.153+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:35:41.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:35:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:35:40.658+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:35:41.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:35:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:35:40.658+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:35:41.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:35:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:35:40.658+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:35:41.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:35:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:35:40.658+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:35:41.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:35:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:35:40.712+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:35:41.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:35:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:35:40.712+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:35:41.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:35:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:35:40.712+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:35:41.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:35:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:35:40.712+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:35:41.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:35:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:35:41.165+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:35:41.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:35:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:35:41.165+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:35:41.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:35:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:35:41.165+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:35:41.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:35:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:35:41.166+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:35:42.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:35:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:35:41.692+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:35:42.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:35:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:35:41.692+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:35:42.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:35:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:35:41.692+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:35:42.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:35:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:35:41.692+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:35:42.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:35:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:35:41.633+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:35:42.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:35:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:35:41.633+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:35:42.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:35:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:35:41.633+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:35:42.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:35:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:35:41.634+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:35:42.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:35:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:35:42.136+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:35:42.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:35:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:35:42.136+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:35:42.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:35:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:35:42.136+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:35:42.571 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:35:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:35:42.137+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:35:43.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:35:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:35:42.632+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:35:43.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:35:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:35:42.632+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:35:43.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:35:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:35:42.632+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:35:43.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:35:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:35:42.633+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:35:43.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:35:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:35:42.727+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:35:43.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:35:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:35:42.727+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:35:43.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:35:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:35:42.727+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:35:43.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:35:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:35:42.728+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:35:43.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:35:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:35:43.137+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:35:43.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:35:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:35:43.137+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:35:43.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:35:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:35:43.137+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:35:43.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:35:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:35:43.138+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:35:44.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:35:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:35:43.624+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:35:44.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:35:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:35:43.624+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:35:44.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:35:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:35:43.624+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:35:44.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:35:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:35:43.625+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:35:44.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:35:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:35:43.696+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:35:44.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:35:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:35:43.696+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:35:44.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:35:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:35:43.696+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:35:44.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:35:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:35:43.697+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:35:44.411 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:35:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:35:44.166+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:35:44.411 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:35:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:35:44.166+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:35:44.411 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:35:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:35:44.166+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:35:44.412 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:35:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:35:44.167+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:35:44.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:35:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:35:44.630+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:35:44.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:35:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:35:44.630+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:35:44.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:35:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:35:44.630+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:35:44.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:35:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:35:44.630+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:35:45.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:35:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:35:44.726+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:35:45.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:35:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:35:44.726+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:35:45.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:35:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:35:44.726+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:35:45.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:35:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:35:44.727+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:35:45.414 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:35:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:35:45.288+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 12982 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:35:45.415 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:35:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:35:45.136+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:35:45.415 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:35:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:35:45.136+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:35:45.415 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:35:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:35:45.136+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:35:45.415 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:35:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:35:45.137+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:35:45.731 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:35:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:35:45.621+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:35:45.731 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:35:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:35:45.621+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:35:45.731 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:35:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:35:45.621+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:35:45.731 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:35:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:35:45.622+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:35:46.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:35:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:35:45.730+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:35:46.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:35:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:35:45.730+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:35:46.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:35:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:35:45.730+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:35:46.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:35:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:35:45.731+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:35:46.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:35:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:35:46.114+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:35:46.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:35:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:35:46.114+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:35:46.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:35:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:35:46.114+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:35:46.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:35:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:35:46.115+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:35:47.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:35:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:35:46.733+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:35:47.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:35:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:35:46.733+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:35:47.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:35:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:35:46.733+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:35:47.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:35:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:35:46.733+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:35:47.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:35:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:35:46.582+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:35:47.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:35:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:35:46.582+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:35:47.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:35:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:35:46.582+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:35:47.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:35:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:35:46.582+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:35:47.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:35:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:35:47.118+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:35:47.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:35:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:35:47.118+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:35:47.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:35:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:35:47.118+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:35:47.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:35:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:35:47.118+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:35:48.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:35:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:35:47.707+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:35:48.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:35:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:35:47.707+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:35:48.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:35:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:35:47.707+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:35:48.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:35:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:35:47.707+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:35:48.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:35:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:35:47.578+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:35:48.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:35:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:35:47.578+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:35:48.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:35:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:35:47.578+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:35:48.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:35:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:35:47.578+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:35:48.531 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:35:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:35:48.120+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:35:48.531 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:35:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:35:48.120+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:35:48.531 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:35:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:35:48.120+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:35:48.531 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:35:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:35:48.121+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:35:48.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:35:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:35:48.751+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:35:48.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:35:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:35:48.751+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:35:48.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:35:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:35:48.751+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:35:48.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:35:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:35:48.752+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:35:48.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:35:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:35:48.529+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:35:48.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:35:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:35:48.529+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:35:48.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:35:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:35:48.529+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:35:48.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:35:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:35:48.530+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:35:49.506 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:35:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:35:49.114+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:35:49.506 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:35:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:35:49.114+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:35:49.507 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:35:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:35:49.114+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:35:49.507 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:35:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:35:49.115+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:35:49.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:35:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:35:49.734+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:35:49.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:35:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:35:49.734+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:35:49.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:35:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:35:49.734+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:35:49.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:35:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:35:49.735+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:35:49.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:35:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:35:49.504+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:35:49.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:35:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:35:49.504+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:35:49.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:35:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:35:49.504+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:35:49.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:35:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:35:49.504+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:35:50.414 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:35:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:35:50.288+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 13075 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:35:50.414 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:35:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:35:50.133+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:35:50.414 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:35:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:35:50.133+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:35:50.415 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:35:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:35:50.133+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:35:50.415 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:35:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:35:50.134+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:35:50.778 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:35:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:35:50.460+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:35:50.779 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:35:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:35:50.460+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:35:50.779 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:35:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:35:50.460+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:35:50.779 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:35:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:35:50.460+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:35:51.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:35:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:35:50.775+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:35:51.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:35:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:35:50.775+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:35:51.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:35:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:35:50.775+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:35:51.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:35:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:35:50.775+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:35:51.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:35:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:35:51.180+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:35:51.478 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:35:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:35:51.180+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:35:51.478 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:35:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:35:51.180+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:35:51.478 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:35:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:35:51.181+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:35:51.814 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:35:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:35:51.476+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:35:51.814 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:35:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:35:51.476+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:35:51.814 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:35:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:35:51.476+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:35:51.814 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:35:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:35:51.476+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:35:52.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:35:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:35:51.812+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:35:52.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:35:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:35:51.812+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:35:52.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:35:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:35:51.812+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:35:52.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:35:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:35:51.813+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:35:52.509 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:35:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:35:52.181+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:35:52.509 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:35:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:35:52.181+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:35:52.509 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:35:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:35:52.181+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:35:52.509 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:35:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:35:52.182+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:35:52.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:35:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:35:52.507+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:35:52.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:35:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:35:52.507+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:35:52.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:35:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:35:52.507+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:35:52.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:35:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:35:52.507+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:35:53.165 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:35:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:35:52.827+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:35:53.166 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:35:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:35:52.827+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:35:53.166 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:35:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:35:52.827+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:35:53.166 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:35:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:35:52.828+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:35:53.514 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:35:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:35:53.163+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:35:53.515 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:35:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:35:53.163+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:35:53.515 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:35:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:35:53.163+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:35:53.515 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:35:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:35:53.164+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:35:53.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:35:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:35:53.512+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:35:53.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:35:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:35:53.512+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:35:53.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:35:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:35:53.512+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:35:53.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:35:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:35:53.513+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:35:54.182 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:35:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:35:53.873+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:35:54.182 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:35:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:35:53.873+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:35:54.182 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:35:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:35:53.873+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:35:54.183 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:35:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:35:53.874+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:35:54.555 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:35:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:35:54.180+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:35:54.555 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:35:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:35:54.180+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:35:54.555 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:35:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:35:54.180+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:35:54.556 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:35:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:35:54.181+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:35:54.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:35:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:35:54.553+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:35:54.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:35:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:35:54.553+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:35:54.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:35:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:35:54.553+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:35:54.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:35:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:35:54.554+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:35:55.192 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:35:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:35:54.903+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:35:55.192 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:35:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:35:54.903+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:35:55.192 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:35:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:35:54.903+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:35:55.192 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:35:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:35:54.904+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:35:55.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:35:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:35:55.290+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 13169 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:35:55.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:35:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:35:55.190+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:35:55.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:35:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:35:55.190+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:35:55.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:35:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:35:55.190+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:35:55.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:35:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:35:55.191+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:35:55.868 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:35:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:35:55.595+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:35:55.868 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:35:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:35:55.595+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:35:55.868 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:35:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:35:55.595+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:35:55.868 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:35:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:35:55.595+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:35:56.202 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:35:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:35:55.866+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:35:56.202 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:35:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:35:55.866+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:35:56.203 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:35:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:35:55.866+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:35:56.203 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:35:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:35:55.867+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:35:56.567 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:35:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:35:56.200+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:35:56.568 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:35:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:35:56.200+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:35:56.568 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:35:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:35:56.200+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:35:56.568 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:35:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:35:56.201+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:35:56.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:35:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:35:56.566+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:35:56.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:35:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:35:56.566+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:35:56.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:35:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:35:56.566+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:35:56.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:35:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:35:56.566+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:35:57.192 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:35:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:35:56.878+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:35:57.193 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:35:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:35:56.878+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:35:57.193 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:35:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:35:56.878+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:35:57.193 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:35:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:35:56.879+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:35:57.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:35:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:35:57.191+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:35:57.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:35:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:35:57.191+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:35:57.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:35:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:35:57.191+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:35:57.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:35:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:35:57.192+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:35:57.901 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:35:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:35:57.568+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:35:57.901 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:35:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:35:57.568+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:35:57.901 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:35:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:35:57.568+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:35:57.901 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:35:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:35:57.569+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:35:58.224 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:35:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:35:57.900+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:35:58.224 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:35:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:35:57.900+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:35:58.224 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:35:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:35:57.900+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:35:58.224 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:35:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:35:57.900+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:35:58.548 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:35:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:35:58.222+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:35:58.548 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:35:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:35:58.222+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:35:58.549 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:35:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:35:58.222+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:35:58.549 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:35:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:35:58.223+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:35:58.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:35:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:35:58.546+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:35:58.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:35:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:35:58.546+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:35:58.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:35:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:35:58.546+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:35:58.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:35:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:35:58.547+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:35:59.115 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:35:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:35:58.858+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:35:59.115 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:35:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:35:58.858+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:35:59.116 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:35:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:35:58.858+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:35:59.116 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:35:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:35:58.859+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:35:59.510 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:35:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:35:59.257+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:35:59.511 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:35:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:35:59.257+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:35:59.511 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:35:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:35:59.257+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:35:59.511 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:35:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:35:59.258+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:35:59.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:35:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:35:59.509+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:35:59.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:35:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:35:59.509+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:35:59.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:35:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:35:59.509+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:35:59.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:35:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:35:59.510+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:36:00.150 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:35:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:35:59.861+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:36:00.151 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:35:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:35:59.861+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:36:00.151 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:35:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:35:59.861+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:36:00.151 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:35:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:35:59.861+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:36:00.415 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:36:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:36:00.292+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 13261 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:36:00.416 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:36:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:36:00.297+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:36:00.416 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:36:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:36:00.297+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:36:00.416 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:36:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:36:00.297+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:36:00.416 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:36:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:36:00.298+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:36:00.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:36:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:36:00.533+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:36:00.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:36:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:36:00.533+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:36:00.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:36:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:36:00.533+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:36:00.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:36:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:36:00.533+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:36:01.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:36:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:36:00.843+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:36:01.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:36:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:36:00.843+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:36:01.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:36:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:36:00.843+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:36:01.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:36:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:36:00.844+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:36:01.684 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:36:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:36:01.327+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:36:01.685 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:36:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:36:01.327+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:36:01.685 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:36:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:36:01.327+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:36:01.685 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:36:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:36:01.327+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:36:01.685 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:36:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:36:01.557+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:36:01.685 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:36:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:36:01.557+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:36:01.686 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:36:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:36:01.557+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:36:01.686 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:36:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:36:01.557+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:36:02.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:36:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:36:01.889+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:36:02.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:36:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:36:01.889+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:36:02.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:36:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:36:01.889+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:36:02.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:36:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:36:01.889+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:36:02.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:36:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:36:02.349+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:36:02.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:36:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:36:02.349+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:36:02.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:36:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:36:02.349+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:36:02.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:36:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:36:02.350+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:36:02.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:36:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:36:02.551+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:36:02.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:36:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:36:02.551+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:36:02.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:36:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:36:02.551+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:36:02.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:36:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:36:02.552+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:36:03.302 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:36:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:36:02.928+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:36:03.302 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:36:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:36:02.928+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:36:03.302 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:36:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:36:02.928+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:36:03.302 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:36:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:36:02.929+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:36:03.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:36:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:36:03.301+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:36:03.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:36:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:36:03.301+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:36:03.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:36:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:36:03.301+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:36:03.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:36:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:36:03.301+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:36:03.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:36:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:36:03.526+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:36:03.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:36:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:36:03.526+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:36:03.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:36:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:36:03.526+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:36:03.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:36:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:36:03.527+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:36:04.259 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:36:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:36:03.897+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:36:04.260 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:36:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:36:03.897+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:36:04.260 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:36:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:36:03.897+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:36:04.261 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:36:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:36:03.898+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:36:04.568 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:36:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:36:04.258+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:36:04.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:36:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:36:04.258+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:36:04.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:36:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:36:04.258+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:36:04.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:36:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:36:04.259+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:36:04.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:36:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:36:04.567+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:36:04.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:36:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:36:04.567+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:36:04.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:36:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:36:04.567+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:36:04.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:36:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:36:04.568+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:36:05.219 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:36:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:36:04.907+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:36:05.220 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:36:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:36:04.907+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:36:05.220 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:36:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:36:04.907+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:36:05.220 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:36:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:36:04.908+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:36:05.555 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:36:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:36:05.294+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 13357 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:36:05.555 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:36:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:36:05.219+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:36:05.555 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:36:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:36:05.219+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:36:05.556 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:36:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:36:05.219+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:36:05.556 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:36:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:36:05.219+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:36:05.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:36:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:36:05.554+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:36:05.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:36:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:36:05.554+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:36:05.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:36:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:36:05.554+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:36:05.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:36:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:36:05.555+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:36:06.243 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:36:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:36:05.948+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:36:06.244 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:36:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:36:05.948+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:36:06.244 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:36:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:36:05.948+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:36:06.244 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:36:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:36:05.949+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:36:06.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:36:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:36:06.243+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:36:06.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:36:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:36:06.243+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:36:06.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:36:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:36:06.243+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:36:06.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:36:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:36:06.243+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:36:06.986 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:36:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:36:06.579+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:36:06.987 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:36:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:36:06.579+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:36:06.987 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:36:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:36:06.579+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:36:06.987 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:36:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:36:06.579+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:36:07.319 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:36:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:36:07.228+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:36:07.319 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:36:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:36:07.228+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:36:07.319 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:36:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:36:07.228+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:36:07.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:36:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:36:07.228+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:36:07.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:36:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:36:06.984+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:36:07.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:36:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:36:06.984+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:36:07.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:36:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:36:06.984+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:36:07.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:36:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:36:06.985+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:36:07.979 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:36:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:36:07.602+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:36:07.979 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:36:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:36:07.602+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:36:07.980 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:36:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:36:07.602+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:36:07.980 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:36:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:36:07.603+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:36:08.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:36:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:36:07.978+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:36:08.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:36:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:36:07.978+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:36:08.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:36:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:36:07.978+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:36:08.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:36:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:36:07.979+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:36:08.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:36:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:36:08.194+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:36:08.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:36:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:36:08.194+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:36:08.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:36:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:36:08.194+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:36:08.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:36:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:36:08.195+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:36:08.953 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:36:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:36:08.608+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:36:08.953 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:36:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:36:08.608+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:36:08.953 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:36:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:36:08.608+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:36:08.953 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:36:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:36:08.608+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:36:09.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:36:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:36:08.952+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:36:09.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:36:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:36:08.952+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:36:09.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:36:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:36:08.952+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:36:09.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:36:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:36:08.953+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:36:09.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:36:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:36:09.191+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:36:09.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:36:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:36:09.191+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:36:09.321 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:36:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:36:09.191+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:36:09.321 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:36:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:36:09.191+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:36:09.973 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:36:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:36:09.586+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:36:09.973 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:36:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:36:09.586+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:36:09.973 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:36:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:36:09.586+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:36:09.974 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:36:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:36:09.586+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:36:10.296 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:36:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:36:10.216+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:36:10.296 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:36:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:36:10.216+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:36:10.296 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:36:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:36:10.216+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:36:10.296 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:36:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:36:10.216+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:36:10.297 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:36:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:36:09.972+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:36:10.297 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:36:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:36:09.972+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:36:10.297 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:36:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:36:09.972+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:36:10.297 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:36:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:36:09.973+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:36:10.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:36:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:36:10.295+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 13451 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:36:10.963 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:36:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:36:10.598+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:36:10.963 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:36:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:36:10.598+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:36:10.964 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:36:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:36:10.598+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:36:10.964 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:36:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:36:10.599+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:36:11.235 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:36:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:36:10.962+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:36:11.235 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:36:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:36:10.962+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:36:11.235 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:36:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:36:10.962+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:36:11.235 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:36:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:36:10.963+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:36:11.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:36:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:36:11.234+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:36:11.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:36:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:36:11.234+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:36:11.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:36:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:36:11.234+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:36:11.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:36:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:36:11.235+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:36:11.932 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:36:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:36:11.626+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:36:11.932 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:36:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:36:11.626+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:36:11.932 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:36:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:36:11.626+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:36:11.932 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:36:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:36:11.627+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:36:12.207 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:36:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:36:11.931+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:36:12.207 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:36:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:36:11.931+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:36:12.207 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:36:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:36:11.931+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:36:12.208 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:36:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:36:11.932+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:36:12.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:36:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:36:12.206+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:36:12.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:36:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:36:12.206+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:36:12.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:36:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:36:12.206+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:36:12.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:36:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:36:12.207+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:36:12.932 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:36:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:36:12.634+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:36:12.932 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:36:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:36:12.634+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:36:12.932 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:36:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:36:12.634+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:36:12.932 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:36:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:36:12.635+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:36:13.206 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:36:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:36:12.931+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:36:13.206 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:36:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:36:12.931+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:36:13.206 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:36:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:36:12.931+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:36:13.206 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:36:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:36:12.932+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:36:13.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:36:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:36:13.205+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:36:13.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:36:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:36:13.205+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:36:13.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:36:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:36:13.205+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:36:13.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:36:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:36:13.205+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:36:13.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:36:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:36:13.665+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:36:13.977 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:36:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:36:13.665+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:36:13.977 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:36:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:36:13.665+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:36:13.977 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:36:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:36:13.665+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:36:14.256 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:36:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:36:13.975+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:36:14.256 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:36:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:36:13.975+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:36:14.256 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:36:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:36:13.975+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:36:14.256 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:36:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:36:13.976+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:36:14.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:36:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:36:14.255+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:36:14.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:36:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:36:14.255+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:36:14.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:36:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:36:14.255+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:36:14.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:36:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:36:14.256+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:36:14.949 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:36:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:36:14.647+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:36:14.949 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:36:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:36:14.647+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:36:14.950 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:36:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:36:14.647+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:36:14.950 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:36:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:36:14.647+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:36:15.287 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:36:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:36:14.948+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:36:15.287 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:36:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:36:14.948+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:36:15.288 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:36:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:36:14.948+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:36:15.288 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:36:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:36:14.949+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:36:15.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:36:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:36:15.296+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 13546 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:36:15.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:36:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:36:15.286+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:36:15.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:36:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:36:15.286+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:36:15.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:36:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:36:15.286+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:36:15.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:36:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:36:15.287+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:36:15.919 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:36:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:36:15.606+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:36:15.920 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:36:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:36:15.606+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:36:15.920 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:36:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:36:15.606+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:36:15.920 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:36:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:36:15.606+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:36:16.249 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:36:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:36:15.918+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:36:16.249 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:36:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:36:15.918+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:36:16.249 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:36:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:36:15.918+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:36:16.249 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:36:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:36:15.919+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:36:16.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:36:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:36:16.248+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:36:16.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:36:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:36:16.248+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:36:16.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:36:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:36:16.248+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:36:16.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:36:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:36:16.249+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:36:16.886 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:36:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:36:16.598+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:36:16.887 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:36:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:36:16.598+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:36:16.887 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:36:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:36:16.598+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:36:16.887 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:36:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:36:16.599+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:36:17.258 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:36:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:36:16.885+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:36:17.258 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:36:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:36:16.885+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:36:17.259 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:36:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:36:16.885+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:36:17.259 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:36:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:36:16.886+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:36:17.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:36:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:36:17.257+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:36:17.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:36:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:36:17.257+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:36:17.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:36:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:36:17.257+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:36:17.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:36:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:36:17.258+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:36:17.904 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:36:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:36:17.590+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:36:17.905 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:36:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:36:17.590+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:36:17.905 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:36:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:36:17.590+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:36:17.905 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:36:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:36:17.591+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:36:18.283 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:36:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:36:17.903+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:36:18.284 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:36:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:36:17.903+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:36:18.284 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:36:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:36:17.903+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:36:18.284 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:36:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:36:17.904+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:36:18.544 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:36:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:36:18.283+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:36:18.544 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:36:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:36:18.283+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:36:18.544 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:36:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:36:18.283+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:36:18.544 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:36:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:36:18.283+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:36:18.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:36:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:36:18.543+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:36:18.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:36:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:36:18.543+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:36:18.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:36:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:36:18.543+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:36:18.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:36:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:36:18.543+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:36:19.272 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:36:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:36:18.866+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:36:19.272 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:36:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:36:18.866+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:36:19.272 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:36:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:36:18.866+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:36:19.273 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:36:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:36:18.867+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:36:19.530 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:36:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:36:19.271+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:36:19.530 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:36:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:36:19.271+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:36:19.530 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:36:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:36:19.271+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:36:19.531 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:36:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:36:19.272+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:36:19.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:36:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:36:19.529+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:36:19.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:36:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:36:19.529+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:36:19.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:36:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:36:19.529+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:36:19.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:36:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:36:19.530+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:36:20.152 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:36:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:36:19.891+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:36:20.152 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:36:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:36:19.891+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:36:20.152 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:36:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:36:19.891+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:36:20.152 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:36:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:36:19.891+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:36:20.421 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:36:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:36:20.298+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 13639 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:36:20.421 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:36:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:36:20.257+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:36:20.422 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:36:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:36:20.257+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:36:20.422 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:36:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:36:20.257+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:36:20.422 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:36:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:36:20.258+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:36:20.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:36:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:36:20.556+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:36:20.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:36:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:36:20.556+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:36:20.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:36:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:36:20.556+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:36:20.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:36:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:36:20.556+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:36:21.234 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:36:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:36:20.910+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:36:21.234 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:36:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:36:20.910+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:36:21.234 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:36:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:36:20.910+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:36:21.234 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:36:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:36:20.911+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:36:21.531 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:36:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:36:21.233+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:36:21.531 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:36:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:36:21.233+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:36:21.531 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:36:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:36:21.233+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:36:21.531 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:36:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:36:21.234+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:36:21.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:36:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:36:21.530+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:36:21.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:36:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:36:21.530+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:36:21.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:36:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:36:21.530+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:36:21.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:36:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:36:21.530+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:36:22.192 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:36:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:36:21.927+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:36:22.192 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:36:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:36:21.927+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:36:22.192 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:36:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:36:21.927+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:36:22.192 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:36:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:36:21.928+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:36:22.543 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:36:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:36:22.191+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:36:22.543 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:36:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:36:22.191+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:36:22.543 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:36:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:36:22.191+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:36:22.543 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:36:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:36:22.192+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:36:22.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:36:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:36:22.542+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:36:22.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:36:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:36:22.542+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:36:22.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:36:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:36:22.542+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:36:22.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:36:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:36:22.543+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:36:23.222 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:36:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:36:22.905+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:36:23.223 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:36:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:36:22.905+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:36:23.223 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:36:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:36:22.905+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:36:23.223 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:36:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:36:22.907+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:36:23.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:36:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:36:23.222+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:36:23.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:36:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:36:23.222+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:36:23.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:36:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:36:23.222+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:36:23.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:36:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:36:23.223+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:36:23.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:36:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:36:23.568+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:36:23.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:36:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:36:23.568+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:36:23.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:36:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:36:23.568+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:36:23.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:36:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:36:23.568+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:36:24.239 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:36:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:36:23.909+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:36:24.239 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:36:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:36:23.909+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:36:24.239 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:36:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:36:23.909+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:36:24.239 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:36:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:36:23.910+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:36:24.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:36:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:36:24.238+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:36:24.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:36:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:36:24.238+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:36:24.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:36:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:36:24.238+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:36:24.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:36:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:36:24.239+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:36:24.913 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:36:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:36:24.606+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:36:24.914 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:36:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:36:24.606+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:36:24.914 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:36:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:36:24.606+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:36:24.914 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:36:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:36:24.607+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:36:25.259 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:36:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:36:24.913+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:36:25.259 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:36:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:36:24.913+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:36:25.259 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:36:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:36:24.913+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:36:25.259 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:36:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:36:24.913+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:36:25.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:36:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:36:25.299+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 13735 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:36:25.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:36:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:36:25.258+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:36:25.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:36:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:36:25.258+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:36:25.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:36:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:36:25.258+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:36:25.571 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:36:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:36:25.259+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:36:25.935 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:36:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:36:25.620+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:36:25.936 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:36:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:36:25.620+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:36:25.936 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:36:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:36:25.620+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:36:25.936 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:36:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:36:25.621+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:36:26.291 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:36:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:36:25.934+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:36:26.292 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:36:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:36:25.934+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:36:26.292 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:36:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:36:25.934+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:36:26.292 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:36:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:36:25.935+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:36:26.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:36:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:36:26.291+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:36:26.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:36:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:36:26.291+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:36:26.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:36:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:36:26.291+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:36:26.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:36:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:36:26.291+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:36:26.968 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:36:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:36:26.623+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:36:26.968 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:36:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:36:26.623+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:36:26.968 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:36:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:36:26.623+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:36:26.968 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:36:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:36:26.624+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:36:27.270 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:36:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:36:26.967+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:36:27.271 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:36:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:36:26.967+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:36:27.271 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:36:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:36:26.967+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:36:27.271 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:36:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:36:26.968+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:36:27.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:36:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:36:27.270+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:36:27.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:36:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:36:27.270+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:36:27.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:36:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:36:27.270+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:36:27.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:36:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:36:27.271+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:36:27.967 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:36:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:36:27.672+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:36:27.967 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:36:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:36:27.672+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:36:27.968 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:36:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:36:27.672+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:36:27.968 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:36:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:36:27.672+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:36:28.256 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:36:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:36:27.966+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:36:28.257 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:36:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:36:27.966+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:36:28.257 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:36:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:36:27.966+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:36:28.257 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:36:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:36:27.967+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:36:28.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:36:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:36:28.255+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:36:28.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:36:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:36:28.255+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:36:28.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:36:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:36:28.255+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:36:28.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:36:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:36:28.256+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:36:29.004 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:36:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:36:28.638+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:36:29.004 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:36:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:36:28.638+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:36:29.005 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:36:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:36:28.638+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:36:29.005 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:36:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:36:28.638+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:36:29.305 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:36:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:36:29.003+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:36:29.305 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:36:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:36:29.003+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:36:29.305 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:36:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:36:29.003+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:36:29.305 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:36:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:36:29.004+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:36:29.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:36:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:36:29.304+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:36:29.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:36:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:36:29.304+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:36:29.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:36:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:36:29.304+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:36:29.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:36:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:36:29.305+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:36:30.047 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:36:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:36:29.620+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:36:30.048 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:36:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:36:29.620+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:36:30.048 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:36:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:36:29.620+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:36:30.048 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:36:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:36:29.620+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:36:30.301 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:36:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:36:30.047+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:36:30.302 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:36:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:36:30.047+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:36:30.302 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:36:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:36:30.047+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:36:30.302 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:36:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:36:30.048+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:36:30.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:36:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:36:30.300+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 13829 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:36:30.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:36:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:36:30.316+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:36:30.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:36:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:36:30.316+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:36:30.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:36:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:36:30.316+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:36:30.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:36:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:36:30.316+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:36:31.016 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:36:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:36:30.661+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:36:31.016 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:36:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:36:30.661+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:36:31.016 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:36:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:36:30.661+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:36:31.016 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:36:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:36:30.661+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:36:31.285 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:36:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:36:31.015+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:36:31.285 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:36:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:36:31.015+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:36:31.285 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:36:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:36:31.015+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:36:31.285 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:36:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:36:31.016+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:36:31.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:36:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:36:31.284+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:36:31.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:36:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:36:31.284+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:36:31.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:36:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:36:31.284+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:36:31.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:36:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:36:31.285+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:36:32.050 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:36:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:36:31.662+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:36:32.051 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:36:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:36:31.662+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:36:32.051 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:36:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:36:31.662+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:36:32.051 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:36:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:36:31.663+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:36:32.319 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:36:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:36:32.255+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:36:32.319 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:36:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:36:32.255+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:36:32.319 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:36:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:36:32.255+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:36:32.319 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:36:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:36:32.255+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:36:32.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:36:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:36:32.049+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:36:32.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:36:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:36:32.049+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:36:32.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:36:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:36:32.049+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:36:32.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:36:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:36:32.050+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:36:33.018 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:36:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:36:32.704+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:36:33.019 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:36:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:36:32.704+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:36:33.019 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:36:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:36:32.704+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:36:33.019 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:36:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:36:32.705+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:36:33.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:36:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:36:33.018+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:36:33.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:36:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:36:33.018+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:36:33.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:36:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:36:33.018+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:36:33.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:36:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:36:33.019+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:36:33.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:36:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:36:33.252+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:36:33.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:36:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:36:33.252+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:36:33.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:36:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:36:33.252+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:36:33.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:36:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:36:33.253+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:36:33.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:36:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:36:33.680+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:36:33.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:36:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:36:33.680+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:36:33.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:36:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:36:33.680+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:36:33.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:36:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:36:33.680+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:36:34.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:36:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:36:34.019+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:36:34.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:36:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:36:34.019+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:36:34.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:36:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:36:34.019+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:36:34.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:36:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:36:34.021+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:36:34.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:36:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:36:34.225+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:36:34.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:36:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:36:34.225+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:36:34.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:36:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:36:34.225+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:36:34.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:36:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:36:34.226+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:36:35.040 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:36:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:36:34.650+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:36:35.041 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:36:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:36:34.650+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:36:35.041 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:36:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:36:34.650+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:36:35.041 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:36:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:36:34.651+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:36:35.303 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:36:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:36:35.039+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:36:35.303 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:36:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:36:35.039+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:36:35.303 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:36:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:36:35.039+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:36:35.303 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:36:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:36:35.041+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:36:35.304 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:36:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:36:35.196+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:36:35.304 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:36:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:36:35.196+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:36:35.304 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:36:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:36:35.196+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:36:35.304 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:36:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:36:35.197+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:36:35.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:36:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:36:35.302+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 13924 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:36:35.997 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:36:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:36:35.699+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:36:35.997 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:36:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:36:35.699+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:36:35.997 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:36:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:36:35.699+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:36:35.997 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:36:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:36:35.700+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:36:36.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:36:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:36:35.996+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:36:36.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:36:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:36:35.996+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:36:36.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:36:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:36:35.996+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:36:36.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:36:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:36:35.997+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:36:36.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:36:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:36:36.184+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:36:36.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:36:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:36:36.184+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:36:36.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:36:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:36:36.184+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:36:36.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:36:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:36:36.185+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:36:36.975 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:36:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:36:36.739+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:36:36.975 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:36:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:36:36.739+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:36:36.975 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:36:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:36:36.739+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:36:36.975 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:36:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:36:36.740+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:36:37.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:36:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:36:36.974+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:36:37.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:36:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:36:36.974+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:36:37.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:36:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:36:36.974+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:36:37.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:36:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:36:36.975+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:36:37.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:36:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:36:37.186+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:36:37.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:36:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:36:37.186+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:36:37.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:36:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:36:37.186+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:36:37.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:36:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:36:37.186+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:36:38.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:36:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:36:37.766+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:36:38.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:36:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:36:37.766+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:36:38.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:36:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:36:37.766+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:36:38.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:36:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:36:37.766+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:36:38.071 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:36:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:36:37.931+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:36:38.071 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:36:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:36:37.931+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:36:38.071 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:36:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:36:37.931+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:36:38.071 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:36:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:36:37.931+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:36:38.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:36:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:36:38.194+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:36:38.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:36:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:36:38.194+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:36:38.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:36:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:36:38.194+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:36:38.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:36:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:36:38.194+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:36:38.977 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:36:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:36:38.724+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:36:38.977 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:36:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:36:38.724+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:36:38.977 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:36:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:36:38.724+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:36:38.978 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:36:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:36:38.725+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:36:39.319 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:36:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:36:39.171+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:36:39.319 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:36:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:36:39.171+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:36:39.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:36:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:36:39.171+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:36:39.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:36:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:36:39.171+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:36:39.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:36:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:36:38.976+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:36:39.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:36:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:36:38.976+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:36:39.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:36:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:36:38.976+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:36:39.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:36:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:36:38.977+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:36:40.003 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:36:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:36:39.746+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:36:40.003 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:36:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:36:39.746+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:36:40.003 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:36:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:36:39.746+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:36:40.003 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:36:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:36:39.747+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:36:40.304 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:36:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:36:40.141+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:36:40.304 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:36:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:36:40.141+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:36:40.305 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:36:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:36:40.141+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:36:40.305 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:36:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:36:40.142+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:36:40.305 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:36:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:36:40.002+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:36:40.305 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:36:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:36:40.002+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:36:40.305 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:36:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:36:40.002+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:36:40.305 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:36:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:36:40.003+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:36:40.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:36:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:36:40.303+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 14017 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:36:41.009 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:36:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:36:40.738+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:36:41.010 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:36:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:36:40.738+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:36:41.010 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:36:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:36:40.738+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:36:41.010 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:36:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:36:40.738+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:36:41.319 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:36:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:36:41.191+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:36:41.319 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:36:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:36:41.191+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:36:41.319 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:36:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:36:41.191+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:36:41.319 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:36:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:36:41.191+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:36:41.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:36:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:36:41.008+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:36:41.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:36:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:36:41.008+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:36:41.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:36:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:36:41.008+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:36:41.321 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:36:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:36:41.009+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:36:41.974 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:36:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:36:41.712+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:36:41.974 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:36:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:36:41.712+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:36:41.975 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:36:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:36:41.712+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:36:41.975 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:36:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:36:41.713+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:36:42.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:36:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:36:41.973+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:36:42.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:36:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:36:41.973+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:36:42.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:36:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:36:41.973+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:36:42.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:36:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:36:41.974+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:36:42.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:36:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:36:42.204+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:36:42.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:36:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:36:42.204+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:36:42.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:36:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:36:42.204+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:36:42.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:36:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:36:42.204+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:36:43.005 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:36:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:36:42.740+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:36:43.006 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:36:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:36:42.740+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:36:43.006 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:36:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:36:42.740+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:36:43.006 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:36:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:36:42.740+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:36:43.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:36:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:36:43.005+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:36:43.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:36:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:36:43.005+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:36:43.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:36:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:36:43.005+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:36:43.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:36:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:36:43.006+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:36:43.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:36:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:36:43.201+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:36:43.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:36:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:36:43.201+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:36:43.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:36:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:36:43.201+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:36:43.321 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:36:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:36:43.202+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:36:44.046 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:36:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:36:43.712+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:36:44.046 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:36:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:36:43.712+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:36:44.046 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:36:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:36:43.712+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:36:44.046 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:36:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:36:43.712+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:36:44.319 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:36:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:36:44.220+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:36:44.319 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:36:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:36:44.220+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:36:44.319 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:36:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:36:44.220+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:36:44.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:36:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:36:44.220+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:36:44.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:36:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:36:44.045+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:36:44.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:36:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:36:44.045+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:36:44.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:36:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:36:44.045+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:36:44.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:36:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:36:44.046+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:36:45.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:36:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:36:44.735+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:36:45.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:36:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:36:44.735+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:36:45.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:36:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:36:44.735+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:36:45.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:36:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:36:44.735+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:36:45.430 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:36:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:36:45.305+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 14112 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:36:45.431 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:36:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:36:45.259+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:36:45.431 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:36:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:36:45.259+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:36:45.431 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:36:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:36:45.259+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:36:45.431 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:36:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:36:45.260+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:36:45.431 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:36:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:36:45.078+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:36:45.431 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:36:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:36:45.078+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:36:45.432 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:36:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:36:45.078+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:36:45.432 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:36:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:36:45.079+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:36:46.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:36:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:36:45.766+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:36:46.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:36:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:36:45.766+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:36:46.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:36:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:36:45.766+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:36:46.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:36:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:36:45.766+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:36:46.569 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:36:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:36:46.087+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:36:46.569 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:36:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:36:46.087+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:36:46.569 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:36:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:36:46.087+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:36:46.570 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:36:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:36:46.088+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:36:46.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:36:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:36:46.221+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:36:46.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:36:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:36:46.221+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:36:46.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:36:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:36:46.221+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:36:46.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:36:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:36:46.222+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:36:47.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:36:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:36:46.753+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:36:47.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:36:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:36:46.753+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:36:47.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:36:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:36:46.753+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:36:47.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:36:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:36:46.754+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:36:47.569 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:36:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:36:47.093+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:36:47.569 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:36:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:36:47.093+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:36:47.570 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:36:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:36:47.093+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:36:47.570 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:36:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:36:47.094+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:36:47.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:36:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:36:47.225+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:36:47.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:36:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:36:47.225+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:36:47.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:36:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:36:47.225+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:36:47.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:36:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:36:47.226+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:36:48.067 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:36:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:36:47.783+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:36:48.068 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:36:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:36:47.783+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:36:48.068 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:36:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:36:47.783+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:36:48.068 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:36:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:36:47.783+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:36:48.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:36:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:36:48.065+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:36:48.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:36:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:36:48.065+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:36:48.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:36:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:36:48.065+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:36:48.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:36:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:36:48.066+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:36:48.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:36:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:36:48.272+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:36:48.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:36:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:36:48.272+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:36:48.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:36:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:36:48.272+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:36:48.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:36:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:36:48.273+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:36:49.115 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:36:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:36:49.050+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:36:49.115 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:36:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:36:49.050+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:36:49.115 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:36:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:36:49.050+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:36:49.115 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:36:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:36:49.051+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:36:49.116 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:36:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:36:48.828+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:36:49.116 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:36:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:36:48.828+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:36:49.116 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:36:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:36:48.828+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:36:49.116 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:36:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:36:48.828+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:36:49.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:36:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:36:49.238+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:36:49.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:36:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:36:49.238+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:36:49.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:36:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:36:49.238+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:36:49.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:36:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:36:49.239+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:36:50.153 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:36:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:36:50.018+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:36:50.154 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:36:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:36:50.018+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:36:50.154 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:36:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:36:50.018+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:36:50.154 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:36:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:36:50.019+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:36:50.154 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:36:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:36:49.830+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:36:50.154 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:36:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:36:49.830+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:36:50.155 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:36:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:36:49.830+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:36:50.155 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:36:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:36:49.830+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:36:50.430 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:36:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:36:50.305+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 14194 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:36:50.431 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:36:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:36:50.215+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:36:50.431 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:36:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:36:50.215+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:36:50.431 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:36:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:36:50.215+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:36:50.431 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:36:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:36:50.216+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:36:51.261 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:36:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:36:51.054+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:36:51.262 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:36:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:36:51.054+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:36:51.262 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:36:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:36:51.054+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:36:51.262 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:36:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:36:51.055+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:36:51.262 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:36:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:36:50.819+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:36:51.262 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:36:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:36:50.819+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:36:51.263 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:36:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:36:50.819+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:36:51.263 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:36:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:36:50.819+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:36:51.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:36:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:36:51.259+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:36:51.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:36:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:36:51.259+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:36:51.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:36:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:36:51.259+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:36:51.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:36:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:36:51.260+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:36:51.994 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:36:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:36:51.864+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:36:51.995 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:36:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:36:51.864+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:36:51.995 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:36:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:36:51.864+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:36:51.995 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:36:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:36:51.865+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:36:52.247 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:36:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:36:52.024+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:36:52.247 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:36:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:36:52.024+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:36:52.247 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:36:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:36:52.024+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:36:52.247 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:36:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:36:52.025+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:36:52.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:36:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:36:52.245+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:36:52.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:36:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:36:52.245+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:36:52.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:36:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:36:52.245+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:36:52.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:36:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:36:52.246+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:36:53.247 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:36:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:36:52.844+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:36:53.247 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:36:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:36:52.844+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:36:53.248 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:36:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:36:52.844+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:36:53.248 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:36:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:36:52.845+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:36:53.248 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:36:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:36:53.006+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:36:53.248 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:36:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:36:53.006+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:36:53.248 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:36:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:36:53.006+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:36:53.248 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:36:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:36:53.007+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:36:53.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:36:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:36:53.245+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:36:53.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:36:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:36:53.245+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:36:53.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:36:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:36:53.245+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:36:53.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:36:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:36:53.246+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:36:54.221 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:36:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:36:54.042+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:36:54.222 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:36:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:36:54.042+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:36:54.222 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:36:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:36:54.042+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:36:54.222 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:36:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:36:54.043+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:36:54.222 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:36:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:36:53.859+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:36:54.222 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:36:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:36:53.859+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:36:54.223 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:36:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:36:53.859+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:36:54.223 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:36:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:36:53.860+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:36:54.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:36:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:36:54.219+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:36:54.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:36:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:36:54.219+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:36:54.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:36:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:36:54.219+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:36:54.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:36:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:36:54.220+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:36:55.208 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:36:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:36:54.896+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:36:55.209 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:36:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:36:54.896+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:36:55.209 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:36:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:36:54.896+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:36:55.209 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:36:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:36:54.896+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:36:55.209 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:36:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:36:55.061+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:36:55.209 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:36:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:36:55.061+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:36:55.210 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:36:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:36:55.061+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:36:55.210 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:36:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:36:55.062+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:36:55.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:36:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:36:55.307+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 14277 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:36:55.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:36:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:36:55.206+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:36:55.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:36:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:36:55.206+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:36:55.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:36:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:36:55.206+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:36:55.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:36:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:36:55.208+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:36:56.252 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:36:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:36:56.047+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:36:56.252 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:36:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:36:56.047+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:36:56.252 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:36:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:36:56.047+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:36:56.252 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:36:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:36:56.048+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:36:56.253 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:36:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:36:55.885+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:36:56.253 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:36:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:36:55.885+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:36:56.253 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:36:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:36:55.885+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:36:56.253 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:36:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:36:55.886+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:36:56.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:36:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:36:56.251+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:36:56.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:36:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:36:56.251+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:36:56.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:36:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:36:56.251+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:36:56.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:36:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:36:56.251+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:36:57.252 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:36:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:36:57.058+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:36:57.253 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:36:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:36:57.058+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:36:57.253 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:36:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:36:57.058+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:36:57.253 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:36:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:36:57.059+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:36:57.254 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:36:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:36:56.881+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:36:57.254 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:36:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:36:56.881+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:36:57.255 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:36:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:36:56.881+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:36:57.255 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:36:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:36:56.881+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:36:57.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:36:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:36:57.251+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:36:57.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:36:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:36:57.251+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:36:57.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:36:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:36:57.251+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:36:57.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:36:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:36:57.251+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:36:58.275 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:36:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:36:58.066+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:36:58.276 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:36:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:36:58.066+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:36:58.276 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:36:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:36:58.066+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:36:58.276 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:36:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:36:58.067+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:36:58.276 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:36:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:36:57.846+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:36:58.276 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:36:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:36:57.846+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:36:58.276 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:36:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:36:57.846+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:36:58.277 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:36:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:36:57.846+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:36:58.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:36:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:36:58.274+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:36:58.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:36:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:36:58.274+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:36:58.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:36:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:36:58.274+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:36:58.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:36:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:36:58.276+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:36:59.116 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:36:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:36:59.032+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:36:59.116 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:36:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:36:59.032+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:36:59.116 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:36:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:36:59.032+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:36:59.116 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:36:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:36:59.033+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:36:59.117 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:36:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:36:58.838+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:36:59.117 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:36:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:36:58.838+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:36:59.117 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:36:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:36:58.838+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:36:59.117 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:36:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:36:58.839+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:36:59.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:36:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:36:59.285+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:36:59.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:36:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:36:59.285+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:36:59.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:36:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:36:59.285+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:36:59.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:36:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:36:59.286+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:37:00.155 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:37:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:37:00.019+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:37:00.155 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:37:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:37:00.019+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:37:00.155 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:37:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:37:00.019+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:37:00.155 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:37:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:37:00.019+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:37:00.155 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:36:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:36:59.821+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:37:00.155 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:36:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:36:59.821+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:37:00.156 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:36:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:36:59.821+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:37:00.156 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:36:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:36:59.821+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:37:00.435 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:37:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:37:00.309+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 14358 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:37:00.436 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:37:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:37:00.311+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:37:00.436 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:37:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:37:00.311+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:37:00.436 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:37:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:37:00.311+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:37:00.436 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:37:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:37:00.312+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:37:01.211 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:37:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:37:00.832+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:37:01.211 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:37:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:37:00.832+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:37:01.211 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:37:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:37:00.832+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:37:01.211 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:37:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:37:00.832+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:37:01.212 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:37:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:37:00.999+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:37:01.212 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:37:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:37:00.999+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:37:01.212 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:37:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:37:00.999+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:37:01.212 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:37:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:37:01.000+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:37:01.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:37:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:37:01.361+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:37:01.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:37:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:37:01.361+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:37:01.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:37:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:37:01.361+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:37:01.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:37:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:37:01.362+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:37:02.046 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:37:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:37:01.835+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:37:02.046 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:37:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:37:01.835+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:37:02.046 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:37:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:37:01.835+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:37:02.046 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:37:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:37:01.836+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:37:02.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:37:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:37:02.044+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:37:02.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:37:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:37:02.044+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:37:02.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:37:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:37:02.044+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:37:02.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:37:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:37:02.045+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:37:02.801 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:37:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:37:02.343+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:37:02.802 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:37:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:37:02.343+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:37:02.802 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:37:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:37:02.343+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:37:02.802 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:37:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:37:02.343+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:37:03.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:37:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:37:02.800+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:37:03.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:37:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:37:02.800+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:37:03.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:37:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:37:02.800+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:37:03.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:37:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:37:02.801+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:37:03.354 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:37:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:37:03.078+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:37:03.354 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:37:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:37:03.078+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:37:03.355 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:37:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:37:03.078+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:37:03.355 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:37:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:37:03.079+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:37:03.766 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:37:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:37:03.351+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:37:03.767 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:37:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:37:03.351+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:37:03.767 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:37:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:37:03.351+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:37:03.767 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:37:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:37:03.352+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:37:04.063 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:37:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:37:03.765+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:37:04.064 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:37:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:37:03.765+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:37:04.064 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:37:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:37:03.765+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:37:04.064 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:37:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:37:03.766+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:37:04.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:37:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:37:04.062+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:37:04.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:37:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:37:04.062+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:37:04.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:37:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:37:04.062+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:37:04.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:37:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:37:04.062+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:37:04.718 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:37:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:37:04.331+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:37:04.719 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:37:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:37:04.331+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:37:04.719 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:37:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:37:04.331+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:37:04.719 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:37:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:37:04.331+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:37:05.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:37:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:37:04.717+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:37:05.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:37:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:37:04.717+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:37:05.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:37:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:37:04.717+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:37:05.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:37:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:37:04.718+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:37:05.436 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:37:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:37:05.311+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 14441 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:37:05.437 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:37:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:37:05.331+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:37:05.437 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:37:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:37:05.331+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:37:05.437 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:37:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:37:05.331+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:37:05.437 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:37:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:37:05.332+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:37:05.438 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:37:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:37:05.101+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:37:05.438 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:37:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:37:05.101+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:37:05.438 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:37:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:37:05.101+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:37:05.438 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:37:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:37:05.102+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:37:06.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:37:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:37:05.718+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:37:06.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:37:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:37:05.718+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:37:06.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:37:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:37:05.718+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:37:06.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:37:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:37:05.719+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:37:06.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:37:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:37:06.342+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:37:06.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:37:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:37:06.342+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:37:06.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:37:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:37:06.342+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:37:06.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:37:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:37:06.343+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:37:06.570 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:37:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:37:06.097+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:37:06.570 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:37:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:37:06.098+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:37:06.570 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:37:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:37:06.098+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:37:06.570 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:37:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:37:06.098+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:37:07.065 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:37:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:37:06.710+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:37:07.065 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:37:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:37:06.710+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:37:07.065 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:37:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:37:06.710+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:37:07.065 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:37:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:37:06.711+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:37:07.317 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:37:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:37:07.064+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:37:07.318 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:37:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:37:07.064+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:37:07.318 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:37:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:37:07.064+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:37:07.318 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:37:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:37:07.065+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:37:07.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:37:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:37:07.316+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:37:07.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:37:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:37:07.316+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:37:07.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:37:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:37:07.316+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:37:07.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:37:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:37:07.317+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:37:08.047 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:37:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:37:07.702+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:37:08.047 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:37:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:37:07.702+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:37:08.047 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:37:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:37:07.702+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:37:08.047 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:37:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:37:07.702+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:37:08.303 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:37:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:37:08.046+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:37:08.303 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:37:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:37:08.046+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:37:08.303 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:37:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:37:08.046+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:37:08.303 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:37:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:37:08.047+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:37:08.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:37:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:37:08.302+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:37:08.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:37:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:37:08.302+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:37:08.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:37:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:37:08.302+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:37:08.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:37:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:37:08.302+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:37:09.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:37:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:37:08.714+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:37:09.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:37:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:37:08.714+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:37:09.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:37:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:37:08.715+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:37:09.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:37:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:37:08.715+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:37:09.569 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:37:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:37:09.092+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:37:09.570 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:37:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:37:09.092+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:37:09.570 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:37:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:37:09.092+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:37:09.570 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:37:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:37:09.092+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:37:09.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:37:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:37:09.265+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:37:09.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:37:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:37:09.265+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:37:09.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:37:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:37:09.265+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:37:09.571 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:37:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:37:09.266+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:37:10.050 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:37:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:37:09.739+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:37:10.050 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:37:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:37:09.739+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:37:10.050 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:37:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:37:09.739+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:37:10.050 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:37:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:37:09.739+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:37:10.314 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:37:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:37:10.286+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:37:10.314 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:37:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:37:10.286+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:37:10.314 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:37:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:37:10.286+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:37:10.314 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:37:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:37:10.287+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:37:10.315 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:37:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:37:10.049+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:37:10.315 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:37:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:37:10.049+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:37:10.315 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:37:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:37:10.049+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:37:10.315 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:37:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:37:10.050+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:37:10.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:37:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:37:10.312+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 14521 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:37:11.063 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:37:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:37:10.695+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:37:11.063 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:37:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:37:10.695+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:37:11.063 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:37:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:37:10.695+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:37:11.063 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:37:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:37:10.696+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:37:11.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:37:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:37:11.061+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:37:11.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:37:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:37:11.061+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:37:11.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:37:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:37:11.061+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:37:11.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:37:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:37:11.062+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:37:11.688 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:37:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:37:11.323+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:37:11.689 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:37:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:37:11.323+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:37:11.689 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:37:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:37:11.323+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:37:11.689 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:37:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:37:11.324+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:37:12.022 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:37:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:37:11.689+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:37:12.022 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:37:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:37:11.689+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:37:12.022 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:37:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:37:11.689+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:37:12.022 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:37:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:37:11.689+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:37:12.286 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:37:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:37:12.021+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:37:12.286 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:37:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:37:12.021+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:37:12.286 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:37:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:37:12.021+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:37:12.287 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:37:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:37:12.022+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:37:12.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:37:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:37:12.285+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:37:12.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:37:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:37:12.285+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:37:12.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:37:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:37:12.285+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:37:12.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:37:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:37:12.286+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:37:13.030 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:37:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:37:12.643+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:37:13.030 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:37:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:37:12.643+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:37:13.030 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:37:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:37:12.643+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:37:13.030 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:37:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:37:12.643+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:37:13.319 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:37:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:37:13.260+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:37:13.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:37:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:37:13.260+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:37:13.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:37:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:37:13.260+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:37:13.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:37:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:37:13.261+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:37:13.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:37:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:37:13.029+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:37:13.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:37:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:37:13.029+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:37:13.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:37:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:37:13.029+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:37:13.321 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:37:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:37:13.030+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:37:13.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:37:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:37:13.645+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:37:13.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:37:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:37:13.645+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:37:13.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:37:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:37:13.645+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:37:13.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:37:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:37:13.646+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:37:14.319 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:37:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:37:14.236+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:37:14.319 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:37:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:37:14.236+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:37:14.319 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:37:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:37:14.236+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:37:14.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:37:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:37:14.237+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:37:14.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:37:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:37:13.987+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:37:14.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:37:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:37:13.987+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:37:14.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:37:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:37:13.987+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:37:14.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:37:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:37:13.988+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:37:14.995 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:37:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:37:14.634+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:37:14.995 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:37:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:37:14.634+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:37:14.995 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:37:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:37:14.634+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:37:14.996 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:37:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:37:14.634+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:37:15.268 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:37:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:37:14.994+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:37:15.268 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:37:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:37:14.994+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:37:15.269 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:37:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:37:14.994+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:37:15.269 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:37:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:37:14.995+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:37:15.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:37:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:37:15.314+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 14603 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:37:15.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:37:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:37:15.267+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:37:15.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:37:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:37:15.267+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:37:15.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:37:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:37:15.267+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:37:15.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:37:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:37:15.268+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:37:16.008 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:37:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:37:15.645+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:37:16.009 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:37:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:37:15.645+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:37:16.009 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:37:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:37:15.645+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:37:16.009 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:37:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:37:15.646+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:37:16.319 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:37:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:37:16.248+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:37:16.319 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:37:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:37:16.248+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:37:16.319 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:37:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:37:16.248+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:37:16.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:37:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:37:16.249+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:37:16.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:37:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:37:16.008+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:37:16.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:37:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:37:16.008+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:37:16.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:37:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:37:16.008+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:37:16.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:37:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:37:16.009+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:37:17.003 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:37:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:37:16.635+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:37:17.003 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:37:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:37:16.635+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:37:17.003 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:37:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:37:16.635+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:37:17.003 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:37:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:37:16.635+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:37:17.268 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:37:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:37:17.002+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:37:17.268 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:37:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:37:17.002+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:37:17.268 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:37:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:37:17.002+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:37:17.268 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:37:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:37:17.003+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:37:17.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:37:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:37:17.267+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:37:17.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:37:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:37:17.267+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:37:17.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:37:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:37:17.267+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:37:17.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:37:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:37:17.268+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:37:18.045 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:37:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:37:17.637+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:37:18.046 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:37:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:37:17.637+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:37:18.046 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:37:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:37:17.637+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:37:18.046 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:37:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:37:17.637+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:37:18.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:37:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:37:18.044+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:37:18.321 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:37:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:37:18.044+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:37:18.321 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:37:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:37:18.044+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:37:18.321 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:37:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:37:18.045+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:37:18.322 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:37:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:37:18.249+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:37:18.322 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:37:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:37:18.249+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:37:18.322 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:37:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:37:18.249+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:37:18.322 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:37:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:37:18.250+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:37:19.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:37:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:37:18.604+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:37:19.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:37:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:37:18.604+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:37:19.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:37:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:37:18.604+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:37:19.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:37:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:37:18.605+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:37:19.569 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:37:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:37:19.068+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:37:19.569 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:37:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:37:19.068+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:37:19.569 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:37:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:37:19.068+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:37:19.570 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:37:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:37:19.069+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:37:19.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:37:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:37:19.207+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:37:19.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:37:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:37:19.208+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:37:19.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:37:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:37:19.208+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:37:19.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:37:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:37:19.208+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:37:19.607 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:37:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:37:19.604+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:37:19.607 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:37:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:37:19.604+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:37:19.607 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:37:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:37:19.604+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:37:19.607 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:37:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:37:19.605+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:37:20.316 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:37:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:37:20.240+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:37:20.317 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:37:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:37:20.240+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:37:20.317 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:37:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:37:20.240+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:37:20.317 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:37:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:37:20.241+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:37:20.317 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:37:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:37:20.050+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:37:20.317 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:37:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:37:20.050+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:37:20.317 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:37:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:37:20.050+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:37:20.318 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:37:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:37:20.051+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:37:20.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:37:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:37:20.315+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 14683 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:37:21.005 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:37:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:37:20.605+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:37:21.006 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:37:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:37:20.605+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:37:21.006 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:37:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:37:20.605+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:37:21.006 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:37:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:37:20.605+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:37:21.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:37:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:37:21.004+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:37:21.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:37:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:37:21.004+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:37:21.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:37:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:37:21.004+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:37:21.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:37:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:37:21.005+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:37:21.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:37:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:37:21.221+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:37:21.321 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:37:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:37:21.221+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:37:21.321 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:37:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:37:21.221+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:37:21.321 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:37:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:37:21.221+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:37:22.024 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:37:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:37:21.610+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:37:22.024 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:37:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:37:21.610+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:37:22.024 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:37:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:37:21.610+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:37:22.024 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:37:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:37:21.611+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:37:22.319 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:37:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:37:22.205+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:37:22.319 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:37:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:37:22.205+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:37:22.319 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:37:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:37:22.205+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:37:22.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:37:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:37:22.206+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:37:22.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:37:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:37:22.023+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:37:22.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:37:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:37:22.023+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:37:22.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:37:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:37:22.023+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:37:22.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:37:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:37:22.024+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:37:23.050 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:37:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:37:22.571+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:37:23.050 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:37:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:37:22.571+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:37:23.051 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:37:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:37:22.571+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:37:23.051 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:37:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:37:22.571+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:37:23.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:37:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:37:23.049+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:37:23.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:37:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:37:23.049+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:37:23.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:37:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:37:23.049+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:37:23.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:37:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:37:23.050+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:37:23.321 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:37:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:37:23.188+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:37:23.321 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:37:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:37:23.188+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:37:23.321 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:37:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:37:23.188+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:37:23.322 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:37:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:37:23.189+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:37:23.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:37:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:37:23.565+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:37:23.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:37:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:37:23.565+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:37:23.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:37:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:37:23.565+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:37:23.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:37:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:37:23.565+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:37:24.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:37:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:37:24.011+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:37:24.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:37:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:37:24.011+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:37:24.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:37:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:37:24.011+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:37:24.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:37:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:37:24.012+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:37:24.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:37:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:37:24.170+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:37:24.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:37:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:37:24.170+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:37:24.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:37:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:37:24.170+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:37:24.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:37:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:37:24.171+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:37:24.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:37:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:37:24.521+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:37:24.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:37:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:37:24.521+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:37:24.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:37:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:37:24.521+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:37:24.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:37:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:37:24.521+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:37:25.318 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:37:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:37:25.174+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:37:25.318 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:37:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:37:25.174+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:37:25.318 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:37:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:37:25.174+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:37:25.318 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:37:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:37:25.175+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:37:25.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:37:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:37:24.982+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:37:25.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:37:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:37:24.982+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:37:25.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:37:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:37:24.982+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:37:25.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:37:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:37:24.983+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:37:25.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:37:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:37:25.317+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 14765 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:37:25.569 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:37:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:37:25.495+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:37:25.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:37:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:37:25.495+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:37:25.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:37:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:37:25.495+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:37:25.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:37:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:37:25.495+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:37:26.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:37:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:37:25.951+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:37:26.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:37:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:37:25.951+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:37:26.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:37:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:37:25.951+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:37:26.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:37:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:37:25.952+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:37:26.485 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:37:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:37:26.170+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:37:26.486 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:37:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:37:26.170+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:37:26.486 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:37:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:37:26.170+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:37:26.486 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:37:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:37:26.171+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:37:26.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:37:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:37:26.484+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:37:26.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:37:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:37:26.484+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:37:26.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:37:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:37:26.484+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:37:26.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:37:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:37:26.485+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:37:27.319 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:37:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:37:27.131+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:37:27.319 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:37:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:37:27.131+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:37:27.319 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:37:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:37:27.131+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:37:27.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:37:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:37:27.132+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:37:27.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:37:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:37:26.986+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:37:27.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:37:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:37:26.987+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:37:27.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:37:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:37:26.987+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:37:27.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:37:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:37:26.987+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:37:27.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:37:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:37:27.444+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:37:27.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:37:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:37:27.444+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:37:27.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:37:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:37:27.444+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:37:27.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:37:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:37:27.444+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:37:28.319 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:37:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:37:28.164+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:37:28.319 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:37:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:37:28.164+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:37:28.319 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:37:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:37:28.164+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:37:28.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:37:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:37:28.165+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:37:28.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:37:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:37:28.030+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:37:28.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:37:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:37:28.030+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:37:28.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:37:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:37:28.030+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:37:28.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:37:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:37:28.031+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:37:28.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:37:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:37:28.415+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:37:28.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:37:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:37:28.415+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:37:28.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:37:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:37:28.415+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:37:28.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:37:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:37:28.416+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:37:29.319 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:37:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:37:29.124+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:37:29.319 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:37:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:37:29.124+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:37:29.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:37:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:37:29.124+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:37:29.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:37:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:37:29.125+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:37:29.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:37:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:37:28.983+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:37:29.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:37:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:37:28.983+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:37:29.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:37:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:37:28.983+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:37:29.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:37:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:37:28.984+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:37:29.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:37:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:37:29.449+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:37:29.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:37:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:37:29.449+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:37:29.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:37:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:37:29.449+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:37:29.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:37:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:37:29.450+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:37:30.319 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:37:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:37:30.152+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:37:30.319 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:37:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:37:30.152+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:37:30.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:37:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:37:30.152+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:37:30.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:37:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:37:30.152+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:37:30.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:37:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:37:29.991+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:37:30.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:37:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:37:29.991+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:37:30.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:37:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:37:29.991+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:37:30.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:37:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:37:29.992+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:37:30.819 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:37:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:37:30.318+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 14846 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:37:30.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:37:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:37:30.439+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:37:30.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:37:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:37:30.439+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:37:30.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:37:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:37:30.439+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:37:30.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:37:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:37:30.439+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:37:31.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:37:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:37:31.024+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:37:31.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:37:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:37:31.024+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:37:31.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:37:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:37:31.025+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:37:31.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:37:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:37:31.025+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:37:31.321 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:37:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:37:31.129+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:37:31.321 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:37:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:37:31.129+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:37:31.321 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:37:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:37:31.129+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:37:31.321 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:37:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:37:31.130+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:37:31.689 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:37:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:37:31.431+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:37:31.690 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:37:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:37:31.431+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:37:31.690 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:37:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:37:31.431+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:37:31.690 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:37:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:37:31.432+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:37:32.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:37:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:37:32.158+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:37:32.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:37:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:37:32.158+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:37:32.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:37:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:37:32.158+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:37:32.321 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:37:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:37:32.158+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:37:32.321 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:37:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:37:32.013+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:37:32.321 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:37:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:37:32.013+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:37:32.322 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:37:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:37:32.013+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:37:32.322 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:37:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:37:32.014+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:37:32.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:37:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:37:32.433+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:37:32.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:37:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:37:32.433+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:37:32.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:37:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:37:32.433+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:37:32.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:37:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:37:32.433+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:37:33.319 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:37:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:37:33.207+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:37:33.319 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:37:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:37:33.207+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:37:33.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:37:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:37:33.207+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:37:33.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:37:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:37:33.208+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:37:33.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:37:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:37:32.974+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:37:33.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:37:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:37:32.974+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:37:33.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:37:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:37:32.974+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:37:33.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:37:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:37:32.975+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:37:33.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:37:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:37:33.403+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:37:33.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:37:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:37:33.403+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:37:33.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:37:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:37:33.403+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:37:33.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:37:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:37:33.404+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:37:34.246 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:37:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:37:33.929+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:37:34.246 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:37:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:37:33.929+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:37:34.246 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:37:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:37:33.929+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:37:34.246 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:37:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:37:33.930+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:37:34.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:37:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:37:34.374+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:37:34.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:37:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:37:34.374+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:37:34.571 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:37:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:37:34.374+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:37:34.571 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:37:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:37:34.375+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:37:34.571 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:37:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:37:34.245+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:37:34.571 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:37:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:37:34.245+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:37:34.571 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:37:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:37:34.245+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:37:34.571 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:37:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:37:34.245+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:37:35.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:37:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:37:34.934+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:37:35.197 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:37:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:37:34.934+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:37:35.197 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:37:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:37:34.934+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:37:35.197 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:37:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:37:34.935+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:37:35.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:37:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:37:35.320+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 14928 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:37:35.569 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:37:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:37:35.400+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:37:35.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:37:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:37:35.400+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:37:35.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:37:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:37:35.400+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:37:35.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:37:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:37:35.400+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:37:35.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:37:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:37:35.195+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:37:35.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:37:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:37:35.195+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:37:35.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:37:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:37:35.195+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:37:35.571 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:37:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:37:35.196+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:37:36.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:37:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:37:35.886+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:37:36.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:37:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:37:35.886+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:37:36.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:37:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:37:35.886+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:37:36.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:37:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:37:35.887+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:37:36.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:37:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:37:36.198+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:37:36.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:37:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:37:36.198+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:37:36.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:37:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:37:36.198+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:37:36.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:37:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:37:36.198+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:37:36.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:37:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:37:36.442+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:37:36.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:37:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:37:36.442+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:37:36.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:37:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:37:36.442+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:37:36.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:37:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:37:36.442+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:37:37.161 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:37:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:37:36.903+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:37:37.161 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:37:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:37:36.903+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:37:37.161 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:37:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:37:36.903+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:37:37.161 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:37:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:37:36.904+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:37:37.413 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:37:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:37:37.160+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:37:37.413 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:37:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:37:37.160+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:37:37.414 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:37:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:37:37.160+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:37:37.414 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:37:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:37:37.161+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:37:37.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:37:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:37:37.412+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:37:37.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:37:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:37:37.412+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:37:37.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:37:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:37:37.412+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:37:37.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:37:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:37:37.412+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:37:38.156 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:37:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:37:37.904+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:37:38.157 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:37:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:37:37.904+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:37:38.157 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:37:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:37:37.904+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:37:38.157 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:37:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:37:37.904+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:37:38.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:37:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:37:38.155+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:37:38.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:37:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:37:38.155+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:37:38.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:37:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:37:38.155+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:37:38.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:37:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:37:38.157+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:37:38.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:37:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:37:38.381+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:37:38.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:37:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:37:38.381+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:37:38.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:37:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:37:38.381+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:37:38.571 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:37:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:37:38.382+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:37:39.205 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:37:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:37:38.884+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:37:39.206 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:37:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:37:38.884+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:37:39.206 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:37:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:37:38.884+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:37:39.206 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:37:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:37:38.885+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:37:39.569 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:37:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:37:39.353+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:37:39.569 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:37:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:37:39.353+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:37:39.569 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:37:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:37:39.353+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:37:39.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:37:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:37:39.354+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:37:39.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:37:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:37:39.204+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:37:39.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:37:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:37:39.204+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:37:39.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:37:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:37:39.204+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:37:39.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:37:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:37:39.205+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:37:40.159 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:37:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:37:39.867+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:37:40.159 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:37:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:37:39.867+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:37:40.159 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:37:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:37:39.867+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:37:40.159 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:37:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:37:39.868+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:37:40.445 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:37:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:37:40.322+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 15008 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:37:40.446 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:37:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:37:40.314+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:37:40.446 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:37:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:37:40.314+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:37:40.446 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:37:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:37:40.314+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:37:40.446 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:37:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:37:40.315+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:37:40.447 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:37:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:37:40.179+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:37:40.447 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:37:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:37:40.179+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:37:40.447 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:37:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:37:40.179+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:37:40.447 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:37:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:37:40.180+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:37:41.147 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:37:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:37:40.872+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:37:41.147 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:37:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:37:40.872+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:37:41.147 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:37:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:37:40.872+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:37:41.147 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:37:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:37:40.873+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:37:41.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:37:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:37:41.149+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:37:41.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:37:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:37:41.149+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:37:41.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:37:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:37:41.149+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:37:41.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:37:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:37:41.150+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:37:41.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:37:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:37:41.344+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:37:41.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:37:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:37:41.344+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:37:41.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:37:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:37:41.344+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:37:41.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:37:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:37:41.344+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:37:42.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:37:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:37:41.858+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:37:42.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:37:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:37:41.858+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:37:42.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:37:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:37:41.858+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:37:42.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:37:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:37:41.859+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:37:42.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:37:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:37:42.157+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:37:42.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:37:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:37:42.157+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:37:42.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:37:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:37:42.157+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:37:42.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:37:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:37:42.158+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:37:42.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:37:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:37:42.312+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:37:42.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:37:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:37:42.312+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:37:42.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:37:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:37:42.312+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:37:42.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:37:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:37:42.312+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:37:43.120 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:37:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:37:42.821+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:37:43.120 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:37:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:37:42.821+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:37:43.120 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:37:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:37:42.821+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:37:43.120 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:37:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:37:42.822+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:37:43.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:37:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:37:43.119+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:37:43.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:37:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:37:43.119+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:37:43.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:37:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:37:43.119+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:37:43.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:37:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:37:43.120+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:37:43.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:37:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:37:43.356+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:37:43.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:37:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:37:43.356+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:37:43.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:37:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:37:43.356+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:37:43.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:37:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:37:43.356+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:37:44.093 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:37:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:37:43.831+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:37:44.093 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:37:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:37:43.831+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:37:44.093 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:37:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:37:43.831+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:37:44.094 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:37:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:37:43.832+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:37:44.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:37:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:37:44.092+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:37:44.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:37:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:37:44.092+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:37:44.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:37:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:37:44.092+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:37:44.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:37:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:37:44.093+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:37:44.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:37:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:37:44.308+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:37:44.574 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:37:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:37:44.308+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:37:44.574 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:37:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:37:44.308+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:37:44.575 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:37:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:37:44.308+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:37:45.141 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:37:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:37:44.850+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:37:45.141 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:37:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:37:44.850+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:37:45.142 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:37:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:37:44.850+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:37:45.142 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:37:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:37:44.851+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:37:45.446 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:37:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:37:45.323+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 15090 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:37:45.447 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:37:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:37:45.140+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:37:45.447 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:37:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:37:45.140+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:37:45.447 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:37:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:37:45.140+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:37:45.447 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:37:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:37:45.141+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:37:45.448 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:37:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:37:45.344+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:37:45.448 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:37:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:37:45.344+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:37:45.448 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:37:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:37:45.344+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:37:45.448 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:37:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:37:45.344+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:37:46.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:37:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:37:45.801+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:37:46.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:37:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:37:45.801+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:37:46.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:37:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:37:45.801+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:37:46.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:37:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:37:45.802+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:37:46.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:37:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:37:46.094+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:37:46.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:37:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:37:46.094+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:37:46.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:37:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:37:46.094+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:37:46.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:37:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:37:46.094+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:37:46.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:37:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:37:46.337+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:37:46.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:37:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:37:46.337+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:37:46.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:37:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:37:46.337+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:37:46.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:37:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:37:46.338+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:37:47.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:37:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:37:46.842+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:37:47.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:37:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:37:46.842+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:37:47.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:37:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:37:46.842+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:37:47.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:37:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:37:46.843+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:37:47.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:37:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:37:47.067+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:37:47.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:37:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:37:47.067+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:37:47.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:37:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:37:47.067+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:37:47.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:37:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:37:47.067+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:37:47.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:37:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:37:47.344+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:37:47.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:37:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:37:47.344+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:37:47.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:37:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:37:47.344+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:37:47.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:37:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:37:47.345+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:37:48.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:37:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:37:47.829+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:37:48.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:37:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:37:47.829+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:37:48.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:37:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:37:47.829+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:37:48.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:37:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:37:47.829+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:37:48.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:37:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:37:48.019+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:37:48.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:37:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:37:48.019+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:37:48.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:37:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:37:48.019+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:37:48.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:37:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:37:48.020+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:37:48.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:37:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:37:48.360+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:37:48.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:37:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:37:48.360+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:37:48.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:37:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:37:48.360+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:37:48.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:37:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:37:48.361+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:37:49.115 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:37:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:37:49.029+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:37:49.115 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:37:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:37:49.029+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:37:49.115 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:37:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:37:49.029+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:37:49.116 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:37:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:37:49.030+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:37:49.116 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:37:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:37:48.827+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:37:49.116 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:37:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:37:48.827+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:37:49.116 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:37:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:37:48.827+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:37:49.116 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:37:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:37:48.828+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:37:49.569 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:37:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:37:49.361+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:37:49.569 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:37:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:37:49.361+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:37:49.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:37:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:37:49.361+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:37:49.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:37:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:37:49.362+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:37:50.159 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:37:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:37:49.838+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:37:50.159 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:37:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:37:49.838+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:37:50.159 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:37:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:37:49.838+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:37:50.160 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:37:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:37:49.839+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:37:50.160 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:37:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:37:50.012+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:37:50.160 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:37:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:37:50.012+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:37:50.160 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:37:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:37:50.012+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:37:50.160 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:37:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:37:50.013+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:37:50.449 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:37:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:37:50.324+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 15170 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:37:50.450 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:37:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:37:50.364+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:37:50.450 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:37:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:37:50.364+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:37:50.450 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:37:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:37:50.364+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:37:50.450 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:37:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:37:50.364+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:37:51.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:37:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:37:50.796+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:37:51.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:37:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:37:50.796+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:37:51.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:37:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:37:50.796+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:37:51.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:37:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:37:50.796+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:37:51.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:37:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:37:50.969+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:37:51.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:37:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:37:50.969+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:37:51.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:37:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:37:50.969+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:37:51.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:37:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:37:50.969+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:37:51.691 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:37:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:37:51.401+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:37:51.691 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:37:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:37:51.401+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:37:51.691 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:37:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:37:51.401+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:37:51.691 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:37:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:37:51.401+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:37:51.965 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:37:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:37:51.770+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:37:51.966 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:37:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:37:51.770+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:37:51.966 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:37:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:37:51.770+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:37:51.966 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:37:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:37:51.771+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:37:52.319 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:37:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:37:51.963+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:37:52.319 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:37:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:37:51.963+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:37:52.319 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:37:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:37:51.963+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:37:52.319 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:37:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:37:51.964+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:37:52.801 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:37:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:37:52.427+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:37:52.802 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:37:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:37:52.427+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:37:52.802 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:37:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:37:52.427+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:37:52.802 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:37:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:37:52.427+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:37:53.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:37:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:37:52.799+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:37:53.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:37:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:37:52.799+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:37:53.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:37:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:37:52.799+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:37:53.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:37:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:37:52.800+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:37:53.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:37:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:37:52.944+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:37:53.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:37:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:37:52.944+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:37:53.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:37:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:37:52.944+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:37:53.072 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:37:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:37:52.945+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:37:53.802 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:37:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:37:53.425+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:37:53.802 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:37:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:37:53.425+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:37:53.802 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:37:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:37:53.425+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:37:53.803 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:37:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:37:53.426+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:37:54.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:37:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:37:53.800+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:37:54.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:37:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:37:53.800+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:37:54.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:37:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:37:53.800+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:37:54.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:37:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:37:53.801+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:37:54.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:37:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:37:53.952+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:37:54.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:37:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:37:53.952+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:37:54.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:37:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:37:53.952+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:37:54.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:37:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:37:53.953+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:37:54.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:37:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:37:54.411+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:37:54.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:37:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:37:54.411+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:37:54.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:37:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:37:54.411+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:37:54.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:37:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:37:54.412+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:37:55.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:37:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:37:54.836+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:37:55.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:37:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:37:54.836+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:37:55.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:37:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:37:54.836+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:37:55.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:37:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:37:54.837+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:37:55.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:37:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:37:54.973+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:37:55.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:37:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:37:54.973+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:37:55.321 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:37:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:37:54.973+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:37:55.321 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:37:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:37:54.975+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:37:55.819 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:37:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:37:55.325+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 15252 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:37:55.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:37:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:37:55.387+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:37:55.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:37:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:37:55.387+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:37:55.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:37:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:37:55.387+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:37:55.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:37:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:37:55.387+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:37:56.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:37:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:37:55.861+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:37:56.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:37:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:37:55.861+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:37:56.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:37:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:37:55.861+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:37:56.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:37:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:37:55.862+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:37:56.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:37:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:37:55.967+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:37:56.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:37:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:37:55.967+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:37:56.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:37:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:37:55.967+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:37:56.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:37:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:37:55.968+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:37:56.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:37:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:37:56.355+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:37:56.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:37:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:37:56.355+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:37:56.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:37:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:37:56.355+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:37:56.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:37:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:37:56.355+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:37:57.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:37:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:37:56.905+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:37:57.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:37:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:37:56.905+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:37:57.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:37:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:37:56.905+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:37:57.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:37:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:37:56.906+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:37:57.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:37:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:37:56.922+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:37:57.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:37:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:37:56.922+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:37:57.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:37:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:37:56.922+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:37:57.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:37:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:37:56.922+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:37:57.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:37:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:37:57.381+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:37:57.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:37:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:37:57.381+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:37:57.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:37:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:37:57.381+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:37:57.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:37:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:37:57.381+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:37:58.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:37:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:37:57.870+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:37:58.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:37:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:37:57.870+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:37:58.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:37:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:37:57.870+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:37:58.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:37:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:37:57.871+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:37:58.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:37:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:37:57.957+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:37:58.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:37:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:37:57.957+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:37:58.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:37:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:37:57.957+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:37:58.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:37:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:37:57.957+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:37:58.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:37:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:37:58.374+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:37:58.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:37:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:37:58.374+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:37:58.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:37:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:37:58.374+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:37:58.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:37:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:37:58.374+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:37:59.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:37:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:37:58.915+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:37:59.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:37:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:37:58.915+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:37:59.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:37:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:37:58.915+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:37:59.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:37:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:37:58.916+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:37:59.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:37:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:37:58.910+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:37:59.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:37:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:37:58.910+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:37:59.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:37:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:37:58.910+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:37:59.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:37:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:37:58.910+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:37:59.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:37:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:37:59.333+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:37:59.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:37:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:37:59.333+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:37:59.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:37:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:37:59.333+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:37:59.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:37:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:37:59.333+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:38:00.299 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:37:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:37:59.937+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:38:00.299 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:37:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:37:59.937+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:38:00.299 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:37:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:37:59.937+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:38:00.300 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:37:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:37:59.938+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:38:00.300 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:37:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:37:59.950+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:38:00.300 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:37:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:37:59.950+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:38:00.300 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:37:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:37:59.950+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:38:00.300 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:37:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:37:59.951+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:38:00.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:38:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:38:00.328+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 15332 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:38:00.569 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:38:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:38:00.298+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:38:00.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:38:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:38:00.298+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:38:00.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:38:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:38:00.298+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:38:00.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:38:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:38:00.299+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:38:01.294 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:38:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:38:00.984+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:38:01.294 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:38:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:38:00.984+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:38:01.294 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:38:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:38:00.984+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:38:01.295 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:38:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:38:00.985+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:38:01.295 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:38:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:38:00.915+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:38:01.295 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:38:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:38:00.915+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:38:01.295 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:38:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:38:00.915+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:38:01.295 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:38:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:38:00.916+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:38:01.569 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:38:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:38:01.293+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:38:01.569 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:38:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:38:01.293+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:38:01.569 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:38:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:38:01.293+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:38:01.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:38:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:38:01.294+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:38:01.978 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:38:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:38:01.902+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:38:01.979 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:38:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:38:01.902+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:38:01.979 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:38:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:38:01.902+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:38:01.979 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:38:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:38:01.903+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:38:02.319 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:38:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:38:01.977+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:38:02.319 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:38:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:38:01.977+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:38:02.319 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:38:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:38:01.977+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:38:02.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:38:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:38:01.978+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:38:02.651 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:38:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:38:02.339+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:38:02.651 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:38:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:38:02.339+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:38:02.651 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:38:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:38:02.339+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:38:02.651 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:38:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:38:02.339+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:38:02.994 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:38:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:38:02.858+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:38:02.994 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:38:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:38:02.858+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:38:02.994 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:38:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:38:02.858+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:38:02.994 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:38:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:38:02.858+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:38:03.319 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:38:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:38:02.993+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:38:03.319 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:38:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:38:02.993+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:38:03.319 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:38:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:38:02.993+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:38:03.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:38:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:38:02.993+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:38:03.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:38:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:38:03.359+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:38:03.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:38:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:38:03.359+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:38:03.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:38:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:38:03.359+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:38:03.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:38:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:38:03.359+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:38:04.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:38:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:38:03.899+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:38:04.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:38:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:38:03.899+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:38:04.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:38:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:38:03.899+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:38:04.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:38:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:38:03.900+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:38:04.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:38:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:38:03.983+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:38:04.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:38:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:38:03.983+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:38:04.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:38:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:38:03.983+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:38:04.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:38:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:38:03.983+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:38:04.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:38:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:38:04.352+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:38:04.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:38:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:38:04.352+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:38:04.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:38:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:38:04.352+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:38:04.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:38:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:38:04.353+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:38:05.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:38:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:38:04.913+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:38:05.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:38:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:38:04.913+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:38:05.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:38:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:38:04.913+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:38:05.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:38:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:38:04.914+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:38:05.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:38:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:38:05.003+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:38:05.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:38:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:38:05.003+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:38:05.321 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:38:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:38:05.003+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:38:05.321 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:38:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:38:05.004+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:38:05.723 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:38:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:38:05.339+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:38:05.723 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:38:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:38:05.339+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:38:05.723 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:38:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:38:05.339+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:38:05.723 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:38:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:38:05.340+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:38:05.723 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:38:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:38:05.330+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 15415 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:38:06.040 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:38:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:38:05.922+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:38:06.041 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:38:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:38:05.922+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:38:06.041 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:38:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:38:05.922+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:38:06.041 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:38:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:38:05.923+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:38:06.319 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:38:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:38:06.040+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:38:06.319 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:38:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:38:06.040+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:38:06.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:38:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:38:06.040+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:38:06.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:38:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:38:06.040+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:38:06.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:38:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:38:06.332+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:38:06.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:38:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:38:06.332+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:38:06.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:38:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:38:06.332+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:38:06.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:38:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:38:06.333+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:38:07.319 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:38:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:38:07.071+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:38:07.319 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:38:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:38:07.071+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:38:07.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:38:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:38:07.071+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:38:07.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:38:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:38:07.071+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:38:07.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:38:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:38:06.943+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:38:07.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:38:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:38:06.943+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:38:07.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:38:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:38:06.943+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:38:07.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:38:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:38:06.944+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:38:07.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:38:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:38:07.360+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:38:07.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:38:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:38:07.360+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:38:07.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:38:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:38:07.360+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:38:07.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:38:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:38:07.361+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:38:08.319 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:38:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:38:08.058+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:38:08.319 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:38:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:38:08.058+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:38:08.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:38:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:38:08.058+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:38:08.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:38:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:38:08.058+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:38:08.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:38:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:38:07.895+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:38:08.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:38:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:38:07.895+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:38:08.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:38:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:38:07.895+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:38:08.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:38:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:38:07.896+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:38:08.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:38:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:38:08.380+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:38:08.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:38:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:38:08.380+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:38:08.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:38:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:38:08.380+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:38:08.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:38:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:38:08.380+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:38:09.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:38:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:38:08.924+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:38:09.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:38:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:38:08.924+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:38:09.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:38:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:38:08.924+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:38:09.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:38:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:38:08.925+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:38:09.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:38:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:38:09.010+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:38:09.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:38:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:38:09.010+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:38:09.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:38:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:38:09.010+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:38:09.321 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:38:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:38:09.011+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:38:09.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:38:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:38:09.398+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:38:09.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:38:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:38:09.398+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:38:09.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:38:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:38:09.398+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:38:09.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:38:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:38:09.399+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:38:10.319 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:38:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:38:10.029+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:38:10.319 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:38:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:38:10.029+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:38:10.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:38:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:38:10.029+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:38:10.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:38:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:38:10.030+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:38:10.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:38:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:38:09.920+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:38:10.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:38:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:38:09.920+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:38:10.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:38:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:38:09.920+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:38:10.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:38:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:38:09.920+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:38:10.819 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:38:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:38:10.331+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 15494 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:38:10.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:38:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:38:10.420+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:38:10.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:38:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:38:10.420+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:38:10.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:38:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:38:10.420+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:38:10.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:38:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:38:10.420+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:38:11.319 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:38:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:38:11.073+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:38:11.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:38:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:38:11.073+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:38:11.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:38:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:38:11.073+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:38:11.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:38:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:38:11.073+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:38:11.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:38:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:38:10.876+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:38:11.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:38:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:38:10.876+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:38:11.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:38:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:38:10.876+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:38:11.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:38:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:38:10.877+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:38:11.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:38:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:38:11.461+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:38:11.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:38:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:38:11.461+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:38:11.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:38:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:38:11.461+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:38:11.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:38:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:38:11.462+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:38:12.319 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:38:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:38:12.066+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:38:12.319 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:38:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:38:12.066+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:38:12.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:38:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:38:12.066+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:38:12.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:38:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:38:12.066+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:38:12.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:38:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:38:11.837+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:38:12.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:38:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:38:11.837+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:38:12.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:38:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:38:11.837+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:38:12.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:38:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:38:11.837+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:38:12.816 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:38:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:38:12.478+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:38:12.816 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:38:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:38:12.479+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:38:12.817 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:38:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:38:12.479+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:38:12.817 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:38:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:38:12.479+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:38:13.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:38:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:38:12.814+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:38:13.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:38:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:38:12.814+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:38:13.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:38:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:38:12.814+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:38:13.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:38:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:38:12.815+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:38:13.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:38:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:38:13.027+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:38:13.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:38:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:38:13.027+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:38:13.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:38:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:38:13.027+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:38:13.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:38:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:38:13.028+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:38:13.809 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:38:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:38:13.463+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:38:13.810 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:38:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:38:13.463+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:38:13.810 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:38:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:38:13.463+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:38:13.810 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:38:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:38:13.463+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:38:14.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:38:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:38:14.018+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:38:14.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:38:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:38:14.018+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:38:14.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:38:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:38:14.018+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:38:14.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:38:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:38:14.019+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:38:14.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:38:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:38:13.808+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:38:14.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:38:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:38:13.808+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:38:14.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:38:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:38:13.808+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:38:14.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:38:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:38:13.809+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:38:14.778 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:38:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:38:14.498+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:38:14.779 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:38:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:38:14.498+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:38:14.779 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:38:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:38:14.498+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:38:14.779 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:38:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:38:14.498+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:38:15.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:38:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:38:15.005+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:38:15.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:38:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:38:15.005+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:38:15.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:38:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:38:15.005+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:38:15.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:38:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:38:15.005+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:38:15.071 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:38:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:38:14.777+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:38:15.071 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:38:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:38:14.777+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:38:15.071 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:38:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:38:14.777+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:38:15.072 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:38:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:38:14.778+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:38:15.819 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:38:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:38:15.333+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 15574 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:38:15.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:38:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:38:15.478+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:38:15.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:38:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:38:15.478+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:38:15.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:38:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:38:15.478+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:38:15.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:38:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:38:15.478+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:38:16.319 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:38:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:38:16.033+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:38:16.319 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:38:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:38:16.033+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:38:16.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:38:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:38:16.033+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:38:16.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:38:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:38:16.033+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:38:16.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:38:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:38:15.822+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:38:16.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:38:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:38:15.822+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:38:16.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:38:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:38:15.822+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:38:16.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:38:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:38:15.822+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:38:16.800 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:38:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:38:16.430+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:38:16.800 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:38:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:38:16.430+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:38:16.800 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:38:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:38:16.430+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:38:16.800 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:38:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:38:16.431+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:38:17.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:38:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:38:16.798+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:38:17.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:38:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:38:16.798+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:38:17.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:38:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:38:16.798+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:38:17.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:38:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:38:16.799+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:38:17.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:38:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:38:17.038+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:38:17.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:38:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:38:17.039+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:38:17.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:38:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:38:17.039+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:38:17.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:38:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:38:17.039+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:38:17.810 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:38:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:38:17.415+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:38:17.810 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:38:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:38:17.415+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:38:17.810 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:38:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:38:17.415+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:38:17.810 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:38:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:38:17.415+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:38:18.068 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:38:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:38:17.809+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:38:18.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:38:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:38:17.809+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:38:18.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:38:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:38:17.809+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:38:18.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:38:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:38:17.809+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:38:18.319 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:38:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:38:18.067+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:38:18.319 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:38:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:38:18.067+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:38:18.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:38:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:38:18.067+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:38:18.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:38:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:38:18.068+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:38:18.812 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:38:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:38:18.367+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:38:18.812 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:38:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:38:18.367+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:38:18.812 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:38:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:38:18.367+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:38:18.812 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:38:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:38:18.368+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:38:19.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:38:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:38:18.811+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:38:19.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:38:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:38:18.811+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:38:19.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:38:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:38:18.811+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:38:19.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:38:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:38:18.812+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:38:19.569 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:38:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:38:19.321+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:38:19.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:38:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:38:19.321+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:38:19.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:38:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:38:19.321+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:38:19.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:38:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:38:19.322+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:38:19.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:38:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:38:19.078+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:38:19.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:38:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:38:19.078+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:38:19.571 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:38:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:38:19.078+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:38:19.571 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:38:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:38:19.079+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:38:20.160 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:38:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:38:20.089+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:38:20.160 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:38:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:38:20.089+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:38:20.160 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:38:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:38:20.089+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:38:20.161 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:38:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:38:20.089+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:38:20.161 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:38:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:38:19.859+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:38:20.161 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:38:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:38:19.859+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:38:20.161 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:38:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:38:19.859+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:38:20.161 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:38:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:38:19.860+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:38:20.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:38:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:38:20.334+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 15654 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:38:20.459 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:38:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:38:20.357+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:38:20.459 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:38:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:38:20.357+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:38:20.459 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:38:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:38:20.357+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:38:20.459 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:38:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:38:20.358+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:38:21.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:38:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:38:20.861+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:38:21.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:38:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:38:20.861+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:38:21.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:38:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:38:20.861+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:38:21.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:38:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:38:20.862+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:38:21.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:38:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:38:21.090+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:38:21.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:38:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:38:21.090+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:38:21.321 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:38:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:38:21.090+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:38:21.321 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:38:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:38:21.090+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:38:21.692 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:38:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:38:21.367+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:38:21.693 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:38:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:38:21.367+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:38:21.693 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:38:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:38:21.367+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:38:21.693 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:38:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:38:21.367+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:38:22.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:38:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:38:21.850+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:38:22.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:38:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:38:21.850+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:38:22.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:38:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:38:21.850+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:38:22.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:38:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:38:21.850+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:38:22.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:38:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:38:22.136+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:38:22.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:38:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:38:22.136+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:38:22.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:38:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:38:22.136+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:38:22.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:38:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:38:22.136+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:38:22.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:38:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:38:22.384+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:38:22.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:38:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:38:22.384+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:38:22.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:38:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:38:22.384+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:38:22.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:38:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:38:22.385+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:38:23.136 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:38:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:38:22.865+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:38:23.137 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:38:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:38:22.865+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:38:23.137 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:38:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:38:22.865+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:38:23.137 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:38:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:38:22.866+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:38:23.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:38:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:38:23.135+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:38:23.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:38:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:38:23.135+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:38:23.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:38:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:38:23.135+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:38:23.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:38:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:38:23.136+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:38:23.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:38:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:38:23.370+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:38:23.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:38:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:38:23.370+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:38:23.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:38:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:38:23.370+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:38:23.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:38:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:38:23.370+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:38:24.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:38:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:38:23.832+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:38:24.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:38:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:38:23.832+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:38:24.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:38:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:38:23.832+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:38:24.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:38:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:38:23.833+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:38:24.569 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:38:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:38:24.365+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:38:24.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:38:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:38:24.365+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:38:24.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:38:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:38:24.365+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:38:24.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:38:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:38:24.365+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:38:24.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:38:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:38:24.162+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:38:24.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:38:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:38:24.162+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:38:24.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:38:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:38:24.162+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:38:24.571 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:38:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:38:24.162+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:38:25.205 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:38:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:38:24.879+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:38:25.205 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:38:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:38:24.879+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:38:25.205 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:38:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:38:24.879+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:38:25.205 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:38:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:38:24.880+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:38:25.462 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:38:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:38:25.336+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 15736 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:38:25.462 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:38:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:38:25.204+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:38:25.463 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:38:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:38:25.204+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:38:25.463 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:38:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:38:25.204+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:38:25.463 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:38:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:38:25.204+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:38:25.463 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:38:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:38:25.396+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:38:25.463 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:38:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:38:25.396+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:38:25.463 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:38:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:38:25.396+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:38:25.463 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:38:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:38:25.396+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:38:26.237 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:38:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:38:25.906+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:38:26.238 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:38:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:38:25.906+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:38:26.238 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:38:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:38:25.906+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:38:26.238 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:38:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:38:25.907+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:38:26.569 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:38:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:38:26.355+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:38:26.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:38:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:38:26.355+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:38:26.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:38:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:38:26.355+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:38:26.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:38:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:38:26.355+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:38:26.573 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:38:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:38:26.236+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:38:26.574 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:38:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:38:26.236+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:38:26.574 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:38:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:38:26.236+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:38:26.574 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:38:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:38:26.237+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:38:27.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:38:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:38:26.945+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:38:27.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:38:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:38:26.945+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:38:27.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:38:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:38:26.945+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:38:27.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:38:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:38:26.946+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:38:27.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:38:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:38:27.193+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:38:27.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:38:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:38:27.193+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:38:27.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:38:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:38:27.193+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:38:27.321 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:38:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:38:27.194+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:38:27.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:38:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:38:27.366+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:38:27.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:38:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:38:27.366+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:38:27.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:38:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:38:27.366+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:38:27.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:38:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:38:27.366+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:38:28.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:38:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:38:27.968+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:38:28.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:38:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:38:27.968+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:38:28.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:38:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:38:27.968+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:38:28.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:38:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:38:27.969+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:38:28.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:38:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:38:28.190+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:38:28.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:38:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:38:28.190+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:38:28.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:38:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:38:28.190+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:38:28.321 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:38:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:38:28.190+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:38:28.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:38:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:38:28.364+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:38:28.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:38:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:38:28.364+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:38:28.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:38:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:38:28.364+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:38:28.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:38:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:38:28.365+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:38:29.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:38:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:38:28.963+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:38:29.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:38:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:38:28.963+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:38:29.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:38:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:38:28.963+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:38:29.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:38:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:38:28.964+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:38:29.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:38:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:38:29.187+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:38:29.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:38:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:38:29.187+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:38:29.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:38:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:38:29.187+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:38:29.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:38:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:38:29.188+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:38:29.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:38:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:38:29.366+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:38:29.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:38:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:38:29.366+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:38:29.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:38:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:38:29.366+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:38:29.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:38:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:38:29.367+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:38:30.299 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:38:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:38:30.150+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:38:30.299 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:38:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:38:30.150+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:38:30.299 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:38:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:38:30.150+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:38:30.299 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:38:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:38:30.151+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:38:30.300 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:38:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:38:29.987+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:38:30.300 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:38:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:38:29.987+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:38:30.300 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:38:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:38:29.987+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:38:30.300 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:38:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:38:29.987+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:38:30.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:38:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:38:30.338+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 15815 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:38:30.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:38:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:38:30.381+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:38:30.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:38:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:38:30.381+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:38:30.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:38:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:38:30.381+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:38:30.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:38:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:38:30.381+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:38:31.319 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:38:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:38:31.167+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:38:31.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:38:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:38:31.167+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:38:31.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:38:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:38:31.167+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:38:31.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:38:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:38:31.167+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:38:31.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:38:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:38:30.984+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:38:31.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:38:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:38:30.984+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:38:31.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:38:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:38:30.984+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:38:31.321 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:38:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:38:30.984+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:38:31.693 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:38:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:38:31.349+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:38:31.694 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:38:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:38:31.349+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:38:31.694 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:38:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:38:31.349+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:38:31.694 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:38:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:38:31.350+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:38:32.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:38:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:38:32.023+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:38:32.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:38:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:38:32.023+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:38:32.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:38:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:38:32.023+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:38:32.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:38:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:38:32.024+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:38:32.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:38:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:38:32.164+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:38:32.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:38:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:38:32.164+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:38:32.321 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:38:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:38:32.164+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:38:32.321 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:38:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:38:32.164+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:38:32.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:38:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:38:32.389+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:38:32.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:38:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:38:32.389+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:38:32.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:38:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:38:32.389+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:38:32.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:38:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:38:32.389+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:38:33.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:38:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:38:32.991+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:38:33.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:38:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:38:32.991+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:38:33.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:38:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:38:32.991+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:38:33.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:38:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:38:32.992+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:38:33.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:38:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:38:33.183+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:38:33.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:38:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:38:33.183+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:38:33.321 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:38:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:38:33.183+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:38:33.321 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:38:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:38:33.184+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:38:33.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:38:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:38:33.340+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:38:33.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:38:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:38:33.340+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:38:33.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:38:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:38:33.340+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:38:33.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:38:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:38:33.340+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:38:34.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:38:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:38:33.954+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:38:34.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:38:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:38:33.954+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:38:34.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:38:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:38:33.954+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:38:34.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:38:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:38:33.955+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:38:34.321 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:38:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:38:34.152+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:38:34.321 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:38:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:38:34.152+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:38:34.321 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:38:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:38:34.152+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:38:34.321 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:38:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:38:34.152+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:38:34.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:38:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:38:34.337+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:38:34.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:38:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:38:34.337+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:38:34.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:38:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:38:34.337+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:38:34.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:38:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:38:34.338+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:38:35.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:38:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:38:34.910+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:38:35.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:38:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:38:34.910+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:38:35.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:38:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:38:34.910+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:38:35.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:38:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:38:34.911+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:38:35.462 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:38:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:38:35.339+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 15897 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:38:35.463 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:38:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:38:35.380+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:38:35.463 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:38:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:38:35.380+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:38:35.463 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:38:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:38:35.380+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:38:35.463 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:38:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:38:35.380+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:38:35.463 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:38:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:38:35.194+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:38:35.463 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:38:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:38:35.194+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:38:35.464 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:38:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:38:35.194+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:38:35.464 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:38:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:38:35.195+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:38:36.188 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:38:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:38:35.915+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:38:36.189 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:38:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:38:35.915+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:38:36.189 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:38:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:38:35.915+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:38:36.189 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:38:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:38:35.916+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:38:36.443 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:38:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:38:36.187+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:38:36.443 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:38:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:38:36.187+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:38:36.443 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:38:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:38:36.187+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:38:36.443 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:38:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:38:36.188+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:38:36.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:38:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:38:36.370+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:38:36.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:38:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:38:36.370+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:38:36.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:38:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:38:36.370+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:38:36.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:38:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:38:36.371+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:38:37.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:38:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:38:36.905+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:38:37.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:38:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:38:36.905+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:38:37.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:38:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:38:36.905+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:38:37.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:38:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:38:36.906+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:38:37.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:38:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:38:37.139+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:38:37.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:38:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:38:37.139+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:38:37.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:38:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:38:37.139+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:38:37.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:38:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:38:37.140+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:38:37.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:38:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:38:37.369+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:38:37.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:38:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:38:37.369+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:38:37.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:38:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:38:37.369+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:38:37.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:38:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:38:37.370+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:38:38.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:38:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:38:37.947+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:38:38.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:38:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:38:37.947+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:38:38.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:38:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:38:37.947+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:38:38.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:38:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:38:37.947+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:38:38.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:38:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:38:38.177+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:38:38.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:38:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:38:38.177+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:38:38.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:38:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:38:38.177+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:38:38.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:38:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:38:38.178+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:38:38.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:38:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:38:38.387+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:38:38.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:38:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:38:38.387+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:38:38.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:38:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:38:38.387+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:38:38.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:38:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:38:38.388+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:38:39.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:38:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:38:38.934+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:38:39.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:38:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:38:38.934+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:38:39.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:38:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:38:38.934+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:38:39.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:38:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:38:38.934+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:38:39.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:38:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:38:39.164+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:38:39.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:38:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:38:39.165+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:38:39.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:38:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:38:39.165+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:38:39.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:38:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:38:39.165+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:38:39.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:38:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:38:39.363+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:38:39.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:38:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:38:39.363+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:38:39.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:38:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:38:39.363+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:38:39.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:38:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:38:39.364+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:38:40.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:38:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:38:39.937+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:38:40.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:38:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:38:39.937+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:38:40.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:38:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:38:39.937+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:38:40.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:38:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:38:39.937+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:38:40.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:38:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:38:40.147+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:38:40.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:38:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:38:40.147+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:38:40.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:38:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:38:40.147+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:38:40.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:38:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:38:40.147+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:38:40.819 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:38:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:38:40.341+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 15975 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:38:40.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:38:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:38:40.340+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:38:40.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:38:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:38:40.340+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:38:40.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:38:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:38:40.340+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:38:40.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:38:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:38:40.341+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:38:41.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:38:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:38:40.975+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:38:41.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:38:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:38:40.975+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:38:41.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:38:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:38:40.975+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:38:41.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:38:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:38:40.976+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:38:41.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:38:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:38:41.116+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:38:41.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:38:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:38:41.116+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:38:41.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:38:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:38:41.116+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:38:41.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:38:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:38:41.118+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:38:41.694 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:38:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:38:41.380+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:38:41.694 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:38:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:38:41.380+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:38:41.694 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:38:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:38:41.380+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:38:41.694 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:38:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:38:41.380+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:38:42.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:38:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:38:41.942+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:38:42.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:38:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:38:41.942+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:38:42.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:38:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:38:41.942+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:38:42.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:38:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:38:41.942+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:38:42.371 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:38:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:38:42.107+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:38:42.372 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:38:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:38:42.107+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:38:42.372 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:38:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:38:42.107+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:38:42.372 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:38:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:38:42.108+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:38:42.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:38:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:38:42.370+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:38:42.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:38:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:38:42.370+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:38:42.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:38:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:38:42.370+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:38:42.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:38:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:38:42.371+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:38:43.319 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:38:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:38:43.089+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:38:43.319 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:38:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:38:43.089+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:38:43.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:38:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:38:43.089+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:38:43.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:38:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:38:43.090+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:38:43.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:38:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:38:42.974+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:38:43.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:38:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:38:42.975+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:38:43.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:38:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:38:42.975+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:38:43.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:38:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:38:42.976+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:38:43.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:38:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:38:43.397+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:38:43.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:38:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:38:43.397+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:38:43.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:38:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:38:43.397+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:38:43.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:38:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:38:43.398+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:38:44.319 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:38:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:38:44.120+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:38:44.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:38:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:38:44.120+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:38:44.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:38:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:38:44.120+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:38:44.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:38:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:38:44.121+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:38:44.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:38:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:38:44.019+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:38:44.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:38:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:38:44.019+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:38:44.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:38:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:38:44.019+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:38:44.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:38:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:38:44.020+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:38:44.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:38:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:38:44.390+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:38:44.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:38:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:38:44.390+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:38:44.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:38:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:38:44.390+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:38:44.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:38:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:38:44.390+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:38:45.319 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:38:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:38:45.105+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:38:45.319 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:38:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:38:45.105+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:38:45.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:38:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:38:45.105+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:38:45.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:38:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:38:45.106+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:38:45.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:38:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:38:45.010+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:38:45.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:38:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:38:45.010+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:38:45.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:38:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:38:45.010+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:38:45.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:38:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:38:45.011+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:38:45.660 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:38:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:38:45.341+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 16055 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:38:45.660 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:38:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:38:45.348+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:38:45.661 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:38:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:38:45.348+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:38:45.661 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:38:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:38:45.348+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:38:45.661 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:38:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:38:45.348+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:38:46.319 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:38:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:38:46.108+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:38:46.319 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:38:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:38:46.108+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:38:46.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:38:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:38:46.108+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:38:46.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:38:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:38:46.109+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:38:46.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:38:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:38:45.968+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:38:46.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:38:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:38:45.968+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:38:46.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:38:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:38:45.968+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:38:46.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:38:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:38:45.968+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:38:46.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:38:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:38:46.394+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:38:46.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:38:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:38:46.394+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:38:46.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:38:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:38:46.394+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:38:46.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:38:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:38:46.395+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:38:47.319 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:38:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:38:47.093+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:38:47.319 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:38:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:38:47.093+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:38:47.319 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:38:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:38:47.093+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:38:47.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:38:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:38:47.094+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:38:47.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:38:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:38:46.980+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:38:47.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:38:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:38:46.980+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:38:47.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:38:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:38:46.980+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:38:47.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:38:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:38:46.981+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:38:47.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:38:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:38:47.361+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:38:47.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:38:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:38:47.361+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:38:47.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:38:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:38:47.361+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:38:47.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:38:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:38:47.361+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:38:48.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:38:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:38:47.974+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:38:48.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:38:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:38:47.974+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:38:48.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:38:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:38:47.974+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:38:48.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:38:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:38:47.975+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:38:48.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:38:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:38:48.128+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:38:48.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:38:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:38:48.128+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:38:48.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:38:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:38:48.128+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:38:48.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:38:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:38:48.129+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:38:48.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:38:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:38:48.330+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:38:48.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:38:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:38:48.330+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:38:48.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:38:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:38:48.330+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:38:48.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:38:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:38:48.330+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:38:49.294 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:38:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:38:49.143+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:38:49.294 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:38:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:38:49.143+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:38:49.295 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:38:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:38:49.143+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:38:49.295 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:38:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:38:49.144+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:38:49.295 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:38:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:38:48.975+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:38:49.295 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:38:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:38:48.975+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:38:49.295 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:38:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:38:48.975+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:38:49.295 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:38:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:38:48.976+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:38:49.569 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:38:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:38:49.292+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:38:49.569 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:38:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:38:49.292+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:38:49.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:38:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:38:49.292+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:38:49.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:38:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:38:49.293+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:38:50.319 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:38:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:38:50.106+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:38:50.319 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:38:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:38:50.106+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:38:50.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:38:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:38:50.106+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:38:50.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:38:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:38:50.106+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:38:50.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:38:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:38:49.936+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:38:50.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:38:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:38:49.936+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:38:50.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:38:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:38:49.936+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:38:50.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:38:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:38:49.936+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:38:50.819 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:38:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:38:50.343+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 16135 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:38:50.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:38:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:38:50.342+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:38:50.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:38:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:38:50.342+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:38:50.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:38:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:38:50.342+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:38:50.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:38:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:38:50.343+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:38:51.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:38:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:38:50.961+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:38:51.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:38:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:38:50.961+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:38:51.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:38:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:38:50.961+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:38:51.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:38:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:38:50.962+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:38:51.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:38:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:38:51.084+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:38:51.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:38:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:38:51.084+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:38:51.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:38:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:38:51.084+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:38:51.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:38:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:38:51.084+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:38:51.695 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:38:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:38:51.391+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:38:51.695 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:38:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:38:51.391+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:38:51.695 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:38:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:38:51.391+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:38:51.695 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:38:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:38:51.392+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:38:52.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:38:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:38:51.991+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:38:52.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:38:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:38:51.991+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:38:52.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:38:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:38:51.991+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:38:52.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:38:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:38:51.992+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:38:52.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:38:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:38:52.080+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:38:52.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:38:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:38:52.080+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:38:52.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:38:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:38:52.080+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:38:52.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:38:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:38:52.080+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:38:52.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:38:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:38:52.438+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:38:52.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:38:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:38:52.438+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:38:52.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:38:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:38:52.438+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:38:52.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:38:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:38:52.438+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:38:53.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:38:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:38:52.973+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:38:53.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:38:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:38:52.973+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:38:53.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:38:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:38:52.973+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:38:53.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:38:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:38:52.973+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:38:53.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:38:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:38:53.031+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:38:53.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:38:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:38:53.031+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:38:53.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:38:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:38:53.031+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:38:53.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:38:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:38:53.032+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:38:53.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:38:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:38:53.484+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:38:53.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:38:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:38:53.484+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:38:53.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:38:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:38:53.484+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:38:53.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:38:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:38:53.484+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:38:54.319 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:38:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:38:53.999+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:38:54.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:38:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:38:53.999+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:38:54.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:38:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:38:53.999+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:38:54.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:38:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:38:54.000+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:38:54.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:38:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:38:54.019+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:38:54.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:38:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:38:54.019+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:38:54.321 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:38:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:38:54.019+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:38:54.321 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:38:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:38:54.020+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:38:54.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:38:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:38:54.436+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:38:54.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:38:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:38:54.436+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:38:54.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:38:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:38:54.436+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:38:54.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:38:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:38:54.436+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:38:55.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:38:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:38:55.049+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:38:55.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:38:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:38:55.049+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:38:55.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:38:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:38:55.049+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:38:55.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:38:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:38:55.050+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:38:55.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:38:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:38:55.002+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:38:55.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:38:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:38:55.002+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:38:55.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:38:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:38:55.002+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:38:55.321 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:38:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:38:55.003+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:38:55.819 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:38:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:38:55.346+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 16215 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:38:55.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:38:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:38:55.406+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:38:55.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:38:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:38:55.406+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:38:55.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:38:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:38:55.406+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:38:55.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:38:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:38:55.406+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:38:56.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:38:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:38:56.080+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:38:56.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:38:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:38:56.080+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:38:56.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:38:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:38:56.080+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:38:56.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:38:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:38:56.080+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:38:56.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:38:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:38:55.968+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:38:56.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:38:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:38:55.968+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:38:56.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:38:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:38:55.968+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:38:56.321 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:38:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:38:55.969+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:38:56.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:38:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:38:56.374+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:38:56.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:38:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:38:56.374+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:38:56.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:38:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:38:56.374+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:38:56.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:38:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:38:56.375+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:38:57.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:38:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:38:57.092+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:38:57.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:38:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:38:57.092+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:38:57.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:38:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:38:57.092+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:38:57.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:38:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:38:57.092+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:38:57.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:38:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:38:56.935+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:38:57.321 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:38:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:38:56.935+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:38:57.321 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:38:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:38:56.935+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:38:57.321 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:38:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:38:56.935+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:38:57.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:38:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:38:57.335+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:38:57.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:38:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:38:57.335+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:38:57.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:38:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:38:57.335+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:38:57.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:38:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:38:57.335+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:38:58.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:38:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:38:58.114+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:38:58.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:38:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:38:58.114+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:38:58.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:38:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:38:58.114+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:38:58.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:38:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:38:58.114+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:38:58.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:38:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:38:57.898+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:38:58.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:38:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:38:57.898+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:38:58.321 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:38:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:38:57.898+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:38:58.321 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:38:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:38:57.899+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:38:58.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:38:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:38:58.366+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:38:58.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:38:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:38:58.366+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:38:58.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:38:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:38:58.366+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:38:58.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:38:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:38:58.367+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:38:59.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:38:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:38:59.073+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:38:59.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:38:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:38:59.073+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:38:59.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:38:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:38:59.073+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:38:59.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:38:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:38:59.074+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:38:59.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:38:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:38:58.884+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:38:59.321 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:38:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:38:58.884+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:38:59.321 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:38:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:38:58.884+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:38:59.321 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:38:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:38:58.884+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:38:59.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:38:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:38:59.331+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:38:59.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:38:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:38:59.331+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:38:59.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:38:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:38:59.331+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:38:59.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:38:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:38:59.331+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:39:00.118 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:38:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:38:59.847+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:39:00.119 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:38:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:38:59.847+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:39:00.119 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:38:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:38:59.847+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:39:00.119 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:38:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:38:59.848+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:39:00.374 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:39:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:39:00.347+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 16295 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:39:00.375 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:39:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:39:00.347+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:39:00.375 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:39:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:39:00.347+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:39:00.375 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:39:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:39:00.347+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:39:00.375 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:39:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:39:00.348+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:39:00.375 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:39:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:39:00.117+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:39:00.375 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:39:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:39:00.117+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:39:00.376 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:39:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:39:00.117+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:39:00.376 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:39:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:39:00.118+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:39:01.089 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:39:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:39:00.827+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:39:01.089 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:39:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:39:00.827+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:39:01.089 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:39:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:39:00.827+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:39:01.089 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:39:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:39:00.827+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:39:01.386 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:39:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:39:01.088+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:39:01.386 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:39:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:39:01.088+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:39:01.386 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:39:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:39:01.088+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:39:01.387 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:39:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:39:01.088+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:39:01.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:39:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:39:01.385+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:39:01.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:39:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:39:01.385+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:39:01.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:39:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:39:01.385+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:39:01.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:39:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:39:01.385+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:39:02.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:39:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:39:01.873+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:39:02.095 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:39:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:39:01.873+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:39:02.095 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:39:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:39:01.873+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:39:02.095 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:39:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:39:01.874+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:39:02.570 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:39:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:39:02.129+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:39:02.570 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:39:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:39:02.129+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:39:02.570 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:39:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:39:02.129+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:39:02.570 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:39:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:39:02.129+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:39:02.571 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:39:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:39:02.374+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:39:02.571 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:39:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:39:02.374+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:39:02.571 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:39:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:39:02.374+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:39:02.571 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:39:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:39:02.375+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:39:03.319 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:39:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:39:02.901+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:39:03.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:39:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:39:02.901+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:39:03.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:39:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:39:02.901+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:39:03.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:39:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:39:02.902+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:39:03.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:39:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:39:03.111+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:39:03.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:39:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:39:03.111+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:39:03.321 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:39:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:39:03.111+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:39:03.321 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:39:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:39:03.112+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:39:03.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:39:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:39:03.408+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:39:03.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:39:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:39:03.408+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:39:03.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:39:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:39:03.408+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:39:03.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:39:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:39:03.408+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:39:04.319 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:39:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:39:03.926+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:39:04.319 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:39:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:39:03.926+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:39:04.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:39:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:39:03.926+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:39:04.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:39:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:39:03.927+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:39:04.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:39:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:39:04.114+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:39:04.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:39:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:39:04.114+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:39:04.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:39:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:39:04.114+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:39:04.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:39:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:39:04.115+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:39:04.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:39:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:39:04.401+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:39:04.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:39:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:39:04.401+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:39:04.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:39:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:39:04.401+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:39:04.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:39:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:39:04.402+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:39:05.155 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:39:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:39:04.899+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:39:05.155 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:39:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:39:04.899+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:39:05.156 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:39:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:39:04.899+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:39:05.156 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:39:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:39:04.900+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:39:05.472 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:39:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:39:05.349+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 16376 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:39:05.473 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:39:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:39:05.154+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:39:05.473 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:39:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:39:05.154+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:39:05.473 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:39:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:39:05.154+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:39:05.473 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:39:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:39:05.155+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:39:05.473 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:39:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:39:05.375+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:39:05.474 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:39:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:39:05.375+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:39:05.474 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:39:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:39:05.375+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:39:05.474 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:39:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:39:05.376+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:39:06.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:39:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:39:06.106+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:39:06.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:39:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:39:06.106+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:39:06.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:39:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:39:06.106+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:39:06.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:39:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:39:06.107+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:39:06.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:39:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:39:05.900+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:39:06.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:39:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:39:05.900+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:39:06.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:39:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:39:05.900+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:39:06.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:39:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:39:05.900+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:39:06.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:39:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:39:06.400+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:39:06.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:39:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:39:06.400+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:39:06.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:39:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:39:06.400+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:39:06.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:39:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:39:06.401+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:39:07.319 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:39:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:39:06.891+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:39:07.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:39:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:39:06.891+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:39:07.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:39:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:39:06.891+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:39:07.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:39:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:39:06.892+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:39:07.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:39:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:39:07.114+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:39:07.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:39:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:39:07.114+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:39:07.321 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:39:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:39:07.114+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:39:07.321 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:39:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:39:07.114+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:39:07.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:39:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:39:07.418+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:39:07.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:39:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:39:07.418+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:39:07.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:39:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:39:07.418+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:39:07.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:39:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:39:07.419+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:39:08.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:39:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:39:08.081+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:39:08.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:39:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:39:08.081+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:39:08.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:39:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:39:08.081+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:39:08.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:39:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:39:08.081+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:39:08.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:39:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:39:07.886+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:39:08.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:39:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:39:07.886+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:39:08.321 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:39:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:39:07.886+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:39:08.321 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:39:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:39:07.886+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:39:08.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:39:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:39:08.451+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:39:08.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:39:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:39:08.451+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:39:08.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:39:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:39:08.451+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:39:08.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:39:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:39:08.452+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:39:09.115 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:39:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:39:08.844+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:39:09.116 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:39:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:39:08.844+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:39:09.116 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:39:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:39:08.844+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:39:09.116 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:39:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:39:08.845+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:39:09.434 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:39:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:39:09.131+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:39:09.434 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:39:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:39:09.131+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:39:09.434 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:39:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:39:09.131+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:39:09.434 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:39:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:39:09.131+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:39:09.797 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:39:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:39:09.433+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:39:09.797 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:39:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:39:09.433+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:39:09.797 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:39:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:39:09.433+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:39:09.797 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:39:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:39:09.433+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:39:10.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:39:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:39:09.795+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:39:10.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:39:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:39:09.795+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:39:10.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:39:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:39:09.795+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:39:10.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:39:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:39:09.797+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:39:10.352 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:39:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:39:10.099+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:39:10.352 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:39:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:39:10.099+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:39:10.352 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:39:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:39:10.099+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:39:10.353 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:39:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:39:10.100+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:39:10.763 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:39:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:39:10.351+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 16457 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:39:10.763 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:39:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:39:10.456+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:39:10.763 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:39:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:39:10.456+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:39:10.764 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:39:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:39:10.456+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:39:10.764 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:39:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:39:10.456+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:39:11.065 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:39:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:39:10.762+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:39:11.065 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:39:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:39:10.762+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:39:11.065 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:39:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:39:10.762+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:39:11.065 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:39:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:39:10.763+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:39:11.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:39:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:39:11.064+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:39:11.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:39:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:39:11.064+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:39:11.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:39:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:39:11.064+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:39:11.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:39:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:39:11.064+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:39:11.802 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:39:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:39:11.486+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:39:11.802 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:39:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:39:11.486+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:39:11.803 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:39:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:39:11.486+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:39:11.803 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:39:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:39:11.487+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:39:12.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:39:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:39:12.041+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:39:12.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:39:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:39:12.041+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:39:12.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:39:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:39:12.041+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:39:12.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:39:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:39:12.042+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:39:12.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:39:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:39:11.801+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:39:12.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:39:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:39:11.801+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:39:12.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:39:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:39:11.801+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:39:12.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:39:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:39:11.802+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:39:12.818 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:39:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:39:12.450+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:39:12.818 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:39:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:39:12.450+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:39:12.818 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:39:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:39:12.450+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:39:12.818 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:39:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:39:12.450+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:39:13.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:39:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:39:12.999+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:39:13.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:39:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:39:12.999+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:39:13.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:39:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:39:12.999+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:39:13.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:39:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:39:13.000+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:39:13.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:39:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:39:12.817+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:39:13.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:39:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:39:12.817+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:39:13.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:39:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:39:12.817+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:39:13.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:39:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:39:12.817+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:39:13.569 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:39:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:39:13.488+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:39:13.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:39:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:39:13.488+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:39:13.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:39:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:39:13.488+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:39:13.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:39:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:39:13.488+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:39:14.028 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:39:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:39:13.802+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:39:14.028 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:39:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:39:13.802+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:39:14.028 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:39:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:39:13.802+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:39:14.029 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:39:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:39:13.802+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:39:14.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:39:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:39:14.026+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:39:14.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:39:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:39:14.026+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:39:14.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:39:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:39:14.026+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:39:14.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:39:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:39:14.027+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:39:14.754 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:39:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:39:14.473+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:39:14.754 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:39:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:39:14.473+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:39:14.754 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:39:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:39:14.473+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:39:14.754 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:39:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:39:14.473+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:39:15.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:39:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:39:14.985+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:39:15.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:39:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:39:14.985+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:39:15.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:39:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:39:14.985+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:39:15.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:39:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:39:14.986+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:39:15.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:39:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:39:14.753+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:39:15.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:39:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:39:14.753+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:39:15.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:39:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:39:14.753+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:39:15.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:39:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:39:14.753+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:39:15.748 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:39:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:39:15.353+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 16537 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:39:15.748 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:39:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:39:15.437+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:39:15.748 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:39:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:39:15.437+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:39:15.748 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:39:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:39:15.437+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:39:15.748 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:39:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:39:15.437+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:39:16.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:39:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:39:15.961+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:39:16.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:39:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:39:15.961+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:39:16.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:39:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:39:15.961+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:39:16.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:39:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:39:15.962+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:39:16.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:39:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:39:15.747+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:39:16.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:39:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:39:15.747+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:39:16.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:39:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:39:15.747+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:39:16.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:39:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:39:15.747+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:39:16.750 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:39:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:39:16.464+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:39:16.750 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:39:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:39:16.464+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:39:16.750 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:39:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:39:16.464+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:39:16.750 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:39:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:39:16.464+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:39:17.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:39:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:39:16.947+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:39:17.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:39:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:39:16.947+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:39:17.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:39:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:39:16.947+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:39:17.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:39:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:39:16.948+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:39:17.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:39:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:39:16.749+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:39:17.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:39:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:39:16.749+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:39:17.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:39:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:39:16.749+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:39:17.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:39:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:39:16.749+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:39:17.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:39:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:39:17.713+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:39:17.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:39:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:39:17.713+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:39:17.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:39:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:39:17.713+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:39:17.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:39:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:39:17.714+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:39:17.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:39:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:39:17.477+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:39:17.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:39:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:39:17.477+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:39:17.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:39:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:39:17.477+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:39:17.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:39:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:39:17.477+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:39:18.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:39:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:39:17.998+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:39:18.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:39:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:39:17.998+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:39:18.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:39:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:39:17.998+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:39:18.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:39:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:39:17.999+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:39:18.754 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:39:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:39:18.484+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:39:18.754 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:39:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:39:18.484+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:39:18.754 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:39:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:39:18.484+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:39:18.754 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:39:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:39:18.485+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:39:19.004 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:39:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:39:19.002+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:39:19.004 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:39:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:39:19.002+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:39:19.005 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:39:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:39:19.002+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:39:19.005 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:39:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:39:19.003+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:39:19.005 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:39:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:39:18.753+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:39:19.005 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:39:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:39:18.753+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:39:19.005 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:39:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:39:18.753+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:39:19.006 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:39:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:39:18.753+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:39:19.804 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:39:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:39:19.508+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:39:19.804 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:39:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:39:19.508+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:39:19.804 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:39:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:39:19.509+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:39:19.804 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:39:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:39:19.509+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:39:20.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:39:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:39:19.803+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:39:20.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:39:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:39:19.803+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:39:20.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:39:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:39:19.803+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:39:20.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:39:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:39:19.803+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:39:20.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:39:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:39:20.046+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:39:20.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:39:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:39:20.046+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:39:20.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:39:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:39:20.046+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:39:20.071 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:39:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:39:20.048+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:39:20.478 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:39:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:39:20.354+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 16617 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:39:20.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:39:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:39:20.555+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:39:20.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:39:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:39:20.555+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:39:20.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:39:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:39:20.555+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:39:20.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:39:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:39:20.556+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:39:21.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:39:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:39:21.056+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:39:21.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:39:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:39:21.056+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:39:21.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:39:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:39:21.056+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:39:21.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:39:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:39:21.056+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:39:21.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:39:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:39:20.828+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:39:21.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:39:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:39:20.828+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:39:21.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:39:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:39:20.828+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:39:21.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:39:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:39:20.829+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:39:22.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:39:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:39:21.809+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:39:22.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:39:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:39:21.809+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:39:22.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:39:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:39:21.809+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:39:22.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:39:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:39:21.810+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:39:22.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:39:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:39:21.595+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:39:22.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:39:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:39:21.595+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:39:22.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:39:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:39:21.595+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:39:22.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:39:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:39:21.595+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:39:22.523 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:39:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:39:22.080+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:39:22.523 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:39:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:39:22.080+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:39:22.523 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:39:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:39:22.080+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:39:22.523 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:39:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:39:22.081+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:39:22.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:39:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:39:22.627+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:39:22.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:39:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:39:22.627+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:39:22.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:39:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:39:22.627+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:39:22.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:39:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:39:22.628+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:39:23.319 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:39:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:39:22.847+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:39:23.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:39:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:39:22.847+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:39:23.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:39:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:39:22.847+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:39:23.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:39:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:39:22.848+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:39:23.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:39:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:39:23.078+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:39:23.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:39:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:39:23.078+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:39:23.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:39:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:39:23.078+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:39:23.321 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:39:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:39:23.079+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:39:23.885 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:39:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:39:23.582+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:39:23.886 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:39:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:39:23.582+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:39:23.886 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:39:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:39:23.582+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:39:23.886 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:39:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:39:23.583+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:39:24.319 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:39:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:39:23.884+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:39:24.319 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:39:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:39:23.884+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:39:24.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:39:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:39:23.884+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:39:24.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:39:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:39:23.885+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:39:24.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:39:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:39:24.074+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:39:24.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:39:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:39:24.074+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:39:24.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:39:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:39:24.074+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:39:24.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:39:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:39:24.075+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:39:24.907 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:39:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:39:24.592+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:39:24.908 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:39:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:39:24.592+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:39:24.908 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:39:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:39:24.592+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:39:24.908 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:39:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:39:24.593+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:39:25.319 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:39:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:39:24.906+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:39:25.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:39:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:39:24.906+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:39:25.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:39:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:39:24.906+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:39:25.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:39:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:39:24.907+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:39:25.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:39:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:39:25.029+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:39:25.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:39:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:39:25.029+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:39:25.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:39:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:39:25.029+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:39:25.321 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:39:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:39:25.030+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:39:25.640 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:39:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:39:25.356+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 16697 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:39:25.917 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:39:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:39:25.639+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:39:25.918 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:39:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:39:25.639+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:39:25.918 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:39:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:39:25.639+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:39:25.918 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:39:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:39:25.639+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:39:26.319 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:39:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:39:25.916+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:39:26.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:39:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:39:25.916+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:39:26.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:39:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:39:25.916+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:39:26.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:39:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:39:25.917+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:39:26.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:39:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:39:26.074+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:39:26.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:39:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:39:26.074+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:39:26.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:39:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:39:26.074+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:39:26.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:39:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:39:26.075+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:39:26.948 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:39:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:39:26.596+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:39:26.949 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:39:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:39:26.596+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:39:26.949 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:39:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:39:26.596+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:39:26.949 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:39:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:39:26.597+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:39:27.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:39:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:39:27.062+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:39:27.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:39:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:39:27.062+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:39:27.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:39:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:39:27.062+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:39:27.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:39:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:39:27.063+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:39:27.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:39:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:39:26.947+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:39:27.321 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:39:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:39:26.947+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:39:27.321 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:39:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:39:26.947+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:39:27.321 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:39:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:39:26.948+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:39:27.946 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:39:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:39:27.621+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:39:27.946 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:39:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:39:27.621+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:39:27.946 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:39:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:39:27.621+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:39:27.946 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:39:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:39:27.622+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:39:28.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:39:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:39:28.057+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:39:28.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:39:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:39:28.057+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:39:28.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:39:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:39:28.057+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:39:28.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:39:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:39:28.058+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:39:28.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:39:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:39:27.945+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:39:28.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:39:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:39:27.945+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:39:28.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:39:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:39:27.945+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:39:28.321 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:39:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:39:27.946+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:39:28.949 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:39:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:39:28.589+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:39:28.950 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:39:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:39:28.589+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:39:28.950 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:39:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:39:28.589+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:39:28.950 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:39:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:39:28.590+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:39:29.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:39:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:39:29.094+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:39:29.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:39:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:39:29.094+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:39:29.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:39:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:39:29.094+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:39:29.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:39:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:39:29.095+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:39:29.321 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:39:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:39:28.948+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:39:29.321 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:39:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:39:28.948+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:39:29.321 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:39:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:39:28.948+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:39:29.321 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:39:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:39:28.949+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:39:29.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:39:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:39:29.553+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:39:29.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:39:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:39:29.553+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:39:29.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:39:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:39:29.553+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:39:29.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:39:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:39:29.553+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:39:30.319 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:39:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:39:29.984+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:39:30.319 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:39:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:39:29.984+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:39:30.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:39:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:39:29.984+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:39:30.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:39:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:39:29.985+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:39:30.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:39:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:39:30.051+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:39:30.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:39:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:39:30.051+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:39:30.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:39:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:39:30.051+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:39:30.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:39:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:39:30.052+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:39:30.820 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:39:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:39:30.358+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 16778 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:39:30.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:39:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:39:30.546+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:39:30.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:39:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:39:30.546+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:39:30.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:39:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:39:30.546+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:39:30.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:39:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:39:30.546+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:39:31.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:39:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:39:31.029+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:39:31.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:39:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:39:31.029+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:39:31.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:39:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:39:31.029+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:39:31.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:39:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:39:31.030+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:39:31.321 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:39:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:39:30.974+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:39:31.321 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:39:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:39:30.974+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:39:31.321 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:39:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:39:30.974+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:39:31.321 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:39:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:39:30.974+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:39:31.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:39:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:39:31.519+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:39:31.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:39:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:39:31.519+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:39:31.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:39:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:39:31.519+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:39:31.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:39:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:39:31.520+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:39:32.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:39:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:39:31.939+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:39:32.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:39:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:39:31.939+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:39:32.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:39:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:39:31.939+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:39:32.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:39:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:39:31.940+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:39:32.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:39:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:39:31.983+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:39:32.321 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:39:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:39:31.983+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:39:32.321 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:39:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:39:31.983+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:39:32.321 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:39:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:39:31.983+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:39:32.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:39:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:39:32.534+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:39:32.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:39:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:39:32.534+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:39:32.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:39:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:39:32.534+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:39:32.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:39:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:39:32.534+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:39:33.319 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:39:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:39:32.950+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:39:33.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:39:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:39:32.950+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:39:33.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:39:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:39:32.950+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:39:33.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:39:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:39:32.951+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:39:33.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:39:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:39:32.944+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:39:33.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:39:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:39:32.944+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:39:33.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:39:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:39:32.944+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:39:33.321 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:39:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:39:32.944+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:39:33.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:39:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:39:33.495+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:39:33.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:39:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:39:33.495+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:39:33.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:39:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:39:33.495+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:39:33.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:39:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:39:33.496+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:39:34.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:39:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:39:33.961+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:39:34.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:39:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:39:33.961+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:39:34.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:39:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:39:33.961+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:39:34.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:39:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:39:33.962+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:39:34.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:39:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:39:33.946+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:39:34.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:39:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:39:33.946+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:39:34.321 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:39:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:39:33.946+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:39:34.321 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:39:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:39:33.946+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:39:34.811 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:39:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:39:34.544+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:39:34.811 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:39:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:39:34.544+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:39:34.811 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:39:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:39:34.544+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:39:34.811 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:39:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:39:34.545+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:39:35.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:39:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:39:35.012+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:39:35.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:39:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:39:35.012+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:39:35.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:39:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:39:35.012+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:39:35.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:39:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:39:35.012+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:39:35.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:39:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:39:34.947+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:39:35.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:39:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:39:34.947+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:39:35.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:39:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:39:34.947+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:39:35.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:39:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:39:34.948+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:39:35.819 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:39:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:39:35.359+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 16859 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:39:35.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:39:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:39:35.574+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:39:35.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:39:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:39:35.574+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:39:35.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:39:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:39:35.574+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:39:35.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:39:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:39:35.575+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:39:36.319 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:39:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:39:35.945+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:39:36.319 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:39:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:39:35.945+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:39:36.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:39:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:39:35.945+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:39:36.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:39:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:39:35.946+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:39:36.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:39:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:39:36.060+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:39:36.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:39:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:39:36.060+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:39:36.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:39:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:39:36.060+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:39:36.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:39:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:39:36.060+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:39:36.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:39:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:39:36.537+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:39:36.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:39:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:39:36.537+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:39:36.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:39:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:39:36.537+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:39:36.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:39:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:39:36.538+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:39:37.319 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:39:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:39:36.971+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:39:37.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:39:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:39:36.971+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:39:37.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:39:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:39:36.971+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:39:37.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:39:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:39:36.972+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:39:37.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:39:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:39:37.018+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:39:37.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:39:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:39:37.018+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:39:37.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:39:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:39:37.018+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:39:37.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:39:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:39:37.019+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:39:38.004 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:39:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:39:37.572+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:39:38.005 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:39:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:39:37.572+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:39:38.005 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:39:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:39:37.572+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:39:38.005 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:39:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:39:37.572+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:39:38.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:39:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:39:38.003+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:39:38.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:39:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:39:38.003+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:39:38.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:39:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:39:38.003+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:39:38.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:39:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:39:38.005+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:39:38.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:39:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:39:38.020+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:39:38.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:39:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:39:38.020+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:39:38.321 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:39:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:39:38.020+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:39:38.321 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:39:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:39:38.021+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:39:39.024 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:39:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:39:38.596+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:39:39.024 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:39:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:39:38.596+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:39:39.024 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:39:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:39:38.596+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:39:39.024 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:39:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:39:38.596+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:39:39.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:39:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:39:39.023+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:39:39.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:39:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:39:39.023+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:39:39.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:39:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:39:39.023+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:39:39.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:39:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:39:39.024+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:39:39.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:39:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:39:39.058+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:39:39.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:39:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:39:39.058+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:39:39.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:39:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:39:39.058+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:39:39.321 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:39:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:39:39.059+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:39:40.024 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:39:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:39:39.641+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:39:40.024 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:39:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:39:39.641+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:39:40.024 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:39:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:39:39.641+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:39:40.025 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:39:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:39:39.642+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:39:40.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:39:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:39:40.023+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:39:40.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:39:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:39:40.023+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:39:40.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:39:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:39:40.023+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:39:40.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:39:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:39:40.024+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:39:40.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:39:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:39:40.035+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:39:40.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:39:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:39:40.035+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:39:40.321 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:39:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:39:40.035+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:39:40.321 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:39:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:39:40.035+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:39:40.679 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:39:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:39:40.361+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 16941 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:39:40.955 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:39:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:39:40.678+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:39:40.955 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:39:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:39:40.678+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:39:40.955 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:39:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:39:40.678+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:39:40.955 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:39:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:39:40.679+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:39:41.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:39:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:39:41.056+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:39:41.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:39:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:39:41.056+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:39:41.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:39:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:39:41.056+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:39:41.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:39:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:39:41.057+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:39:41.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:39:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:39:41.059+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:39:41.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:39:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:39:41.059+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:39:41.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:39:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:39:41.059+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:39:41.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:39:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:39:41.060+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:39:42.032 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:39:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:39:41.696+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:39:42.032 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:39:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:39:41.696+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:39:42.033 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:39:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:39:41.696+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:39:42.033 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:39:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:39:41.697+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:39:42.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:39:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:39:42.030+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:39:42.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:39:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:39:42.030+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:39:42.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:39:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:39:42.030+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:39:42.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:39:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:39:42.031+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:39:42.321 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:39:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:39:42.071+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:39:42.321 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:39:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:39:42.071+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:39:42.321 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:39:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:39:42.071+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:39:42.321 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:39:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:39:42.072+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:39:42.983 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:39:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:39:42.726+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:39:42.984 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:39:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:39:42.726+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:39:42.984 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:39:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:39:42.726+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:39:42.984 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:39:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:39:42.727+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:39:43.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:39:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:39:42.981+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:39:43.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:39:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:39:42.981+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:39:43.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:39:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:39:42.981+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:39:43.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:39:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:39:42.982+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:39:43.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:39:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:39:43.033+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:39:43.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:39:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:39:43.033+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:39:43.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:39:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:39:43.033+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:39:43.321 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:39:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:39:43.034+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:39:43.960 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:39:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:39:43.777+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:39:43.960 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:39:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:39:43.777+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:39:43.960 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:39:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:39:43.777+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:39:43.960 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:39:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:39:43.777+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:39:44.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:39:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:39:43.958+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:39:44.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:39:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:39:43.958+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:39:44.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:39:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:39:43.958+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:39:44.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:39:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:39:43.958+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:39:44.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:39:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:39:44.005+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:39:44.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:39:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:39:44.005+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:39:44.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:39:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:39:44.005+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:39:44.321 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:39:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:39:44.005+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:39:45.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:39:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:39:44.982+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:39:45.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:39:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:39:44.982+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:39:45.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:39:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:39:44.982+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:39:45.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:39:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:39:44.983+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:39:45.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:39:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:39:44.933+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:39:45.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:39:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:39:44.933+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:39:45.071 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:39:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:39:44.933+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:39:45.071 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:39:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:39:44.933+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:39:45.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:39:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:39:44.761+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:39:45.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:39:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:39:44.761+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:39:45.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:39:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:39:44.761+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:39:45.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:39:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:39:44.761+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:39:45.776 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:39:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:39:45.362+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 17022 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:39:46.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:39:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:39:45.887+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:39:46.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:39:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:39:45.887+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:39:46.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:39:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:39:45.887+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:39:46.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:39:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:39:45.887+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:39:46.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:39:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:39:45.983+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:39:46.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:39:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:39:45.983+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:39:46.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:39:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:39:45.983+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:39:46.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:39:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:39:45.984+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:39:46.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:39:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:39:45.774+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:39:46.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:39:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:39:45.774+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:39:46.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:39:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:39:45.774+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:39:46.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:39:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:39:45.775+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:39:47.002 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:39:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:39:46.864+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:39:47.002 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:39:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:39:46.864+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:39:47.002 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:39:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:39:46.864+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:39:47.002 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:39:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:39:46.865+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:39:47.003 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:39:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:39:46.747+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:39:47.003 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:39:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:39:46.747+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:39:47.003 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:39:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:39:46.747+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:39:47.003 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:39:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:39:46.748+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:39:47.319 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:39:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:39:47.000+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:39:47.319 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:39:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:39:47.000+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:39:47.319 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:39:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:39:47.000+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:39:47.319 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:39:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:39:47.001+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:39:48.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:39:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:39:47.844+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:39:48.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:39:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:39:47.844+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:39:48.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:39:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:39:47.844+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:39:48.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:39:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:39:47.845+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:39:48.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:39:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:39:47.955+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:39:48.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:39:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:39:47.955+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:39:48.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:39:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:39:47.955+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:39:48.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:39:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:39:47.956+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:39:48.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:39:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:39:47.711+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:39:48.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:39:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:39:47.711+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:39:48.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:39:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:39:47.711+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:39:48.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:39:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:39:47.711+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:39:48.857 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:39:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:39:48.700+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:39:48.857 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:39:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:39:48.700+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:39:48.857 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:39:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:39:48.700+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:39:48.858 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:39:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:39:48.700+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:39:49.115 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:39:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:39:48.947+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:39:49.116 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:39:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:39:48.947+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:39:49.116 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:39:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:39:48.947+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:39:49.116 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:39:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:39:48.948+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:39:49.116 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:39:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:39:48.875+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:39:49.116 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:39:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:39:48.875+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:39:49.117 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:39:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:39:48.875+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:39:49.117 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:39:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:39:48.876+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:39:50.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:39:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:39:49.856+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:39:50.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:39:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:39:49.856+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:39:50.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:39:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:39:49.856+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:39:50.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:39:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:39:49.856+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:39:50.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:39:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:39:49.955+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:39:50.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:39:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:39:49.955+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:39:50.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:39:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:39:49.955+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:39:50.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:39:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:39:49.956+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:39:50.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:39:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:39:49.712+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:39:50.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:39:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:39:49.712+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:39:50.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:39:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:39:49.712+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:39:50.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:39:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:39:49.713+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:39:50.489 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:39:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:39:50.364+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 17104 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:39:50.810 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:39:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:39:50.725+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:39:50.810 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:39:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:39:50.725+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:39:50.810 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:39:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:39:50.725+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:39:50.811 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:39:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:39:50.725+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:39:51.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:39:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:39:50.972+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:39:51.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:39:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:39:50.972+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:39:51.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:39:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:39:50.972+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:39:51.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:39:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:39:50.972+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:39:51.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:39:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:39:50.809+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:39:51.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:39:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:39:50.809+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:39:51.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:39:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:39:50.809+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:39:51.071 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:39:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:39:50.809+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:39:51.985 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:39:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:39:51.849+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:39:51.985 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:39:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:39:51.849+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:39:51.986 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:39:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:39:51.849+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:39:51.986 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:39:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:39:51.849+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:39:51.986 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:39:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:39:51.758+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:39:51.986 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:39:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:39:51.758+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:39:51.986 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:39:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:39:51.758+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:39:51.986 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:39:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:39:51.758+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:39:52.319 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:39:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:39:51.984+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:39:52.319 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:39:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:39:51.984+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:39:52.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:39:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:39:51.984+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:39:52.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:39:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:39:51.985+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:39:53.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:39:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:39:52.784+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:39:53.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:39:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:39:52.784+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:39:53.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:39:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:39:52.784+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:39:53.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:39:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:39:52.784+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:39:53.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:39:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:39:52.839+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:39:53.071 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:39:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:39:52.839+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:39:53.071 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:39:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:39:52.839+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:39:53.071 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:39:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:39:52.840+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:39:53.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:39:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:39:52.975+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:39:53.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:39:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:39:52.975+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:39:53.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:39:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:39:52.975+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:39:53.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:39:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:39:52.976+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:39:53.972 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:39:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:39:53.750+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:39:53.973 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:39:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:39:53.750+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:39:53.973 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:39:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:39:53.750+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:39:53.973 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:39:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:39:53.750+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:39:53.973 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:39:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:39:53.819+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:39:53.973 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:39:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:39:53.819+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:39:53.973 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:39:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:39:53.819+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:39:53.974 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:39:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:39:53.819+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:39:54.319 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:39:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:39:53.971+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:39:54.319 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:39:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:39:53.971+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:39:54.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:39:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:39:53.971+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:39:54.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:39:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:39:53.972+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:39:55.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:39:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:39:54.865+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:39:55.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:39:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:39:54.865+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:39:55.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:39:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:39:54.865+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:39:55.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:39:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:39:54.866+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:39:55.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:39:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:39:54.971+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:39:55.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:39:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:39:54.971+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:39:55.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:39:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:39:54.971+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:39:55.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:39:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:39:54.972+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:39:55.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:39:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:39:54.779+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:39:55.072 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:39:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:39:54.780+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:39:55.072 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:39:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:39:54.780+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:39:55.072 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:39:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:39:54.780+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:39:55.772 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:39:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:39:55.366+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 17186 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:39:56.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:39:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:39:55.895+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:39:56.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:39:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:39:55.895+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:39:56.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:39:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:39:55.895+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:39:56.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:39:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:39:55.895+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:39:56.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:39:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:39:55.927+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:39:56.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:39:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:39:55.927+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:39:56.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:39:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:39:55.927+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:39:56.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:39:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:39:55.928+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:39:56.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:39:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:39:55.771+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:39:56.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:39:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:39:55.771+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:39:56.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:39:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:39:55.771+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:39:56.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:39:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:39:55.772+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:39:56.912 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:39:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:39:56.784+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:39:56.912 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:39:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:39:56.784+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:39:56.912 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:39:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:39:56.784+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:39:56.913 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:39:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:39:56.784+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:39:57.255 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:39:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:39:56.911+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:39:57.255 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:39:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:39:56.911+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:39:57.255 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:39:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:39:56.911+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:39:57.255 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:39:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:39:56.912+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:39:57.256 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:39:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:39:56.924+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:39:57.256 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:39:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:39:56.924+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:39:57.256 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:39:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:39:56.924+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:39:57.256 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:39:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:39:56.925+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:39:57.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:39:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:39:57.748+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:39:57.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:39:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:39:57.748+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:39:57.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:39:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:39:57.748+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:39:57.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:39:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:39:57.748+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:39:58.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:39:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:39:57.878+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:39:58.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:39:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:39:57.878+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:39:58.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:39:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:39:57.878+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:39:58.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:39:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:39:57.879+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:39:58.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:39:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:39:57.925+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:39:58.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:39:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:39:57.925+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:39:58.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:39:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:39:57.925+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:39:58.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:39:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:39:57.925+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:39:59.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:39:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:39:58.719+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:39:59.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:39:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:39:58.719+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:39:59.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:39:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:39:58.719+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:39:59.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:39:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:39:58.720+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:39:59.071 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:39:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:39:58.841+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:39:59.071 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:39:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:39:58.841+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:39:59.071 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:39:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:39:58.841+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:39:59.071 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:39:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:39:58.842+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:39:59.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:39:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:39:58.888+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:39:59.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:39:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:39:58.888+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:39:59.072 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:39:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:39:58.888+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:39:59.072 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:39:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:39:58.889+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:40:00.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:39:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:39:59.729+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:40:00.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:39:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:39:59.729+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:40:00.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:39:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:39:59.729+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:40:00.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:39:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:39:59.730+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:40:00.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:39:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:39:59.839+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:40:00.071 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:39:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:39:59.839+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:40:00.071 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:39:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:39:59.839+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:40:00.071 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:39:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:39:59.839+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:40:00.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:39:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:39:59.849+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:40:00.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:39:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:39:59.849+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:40:00.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:39:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:39:59.849+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:40:00.072 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:39:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:39:59.850+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:40:00.490 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:40:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:40:00.368+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 17268 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:40:00.806 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:40:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:40:00.716+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:40:00.806 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:40:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:40:00.716+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:40:00.806 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:40:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:40:00.716+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:40:00.806 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:40:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:40:00.717+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:40:01.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:40:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:40:00.804+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:40:01.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:40:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:40:00.804+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:40:01.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:40:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:40:00.804+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:40:01.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:40:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:40:00.806+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:40:01.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:40:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:40:00.888+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:40:01.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:40:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:40:00.888+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:40:01.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:40:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:40:00.888+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:40:01.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:40:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:40:00.889+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:40:02.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:40:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:40:01.802+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:40:02.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:40:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:40:01.802+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:40:02.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:40:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:40:01.802+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:40:02.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:40:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:40:01.803+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:40:02.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:40:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:40:01.757+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:40:02.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:40:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:40:01.757+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:40:02.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:40:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:40:01.757+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:40:02.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:40:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:40:01.758+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:40:02.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:40:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:40:01.882+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:40:02.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:40:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:40:01.882+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:40:02.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:40:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:40:01.882+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:40:02.072 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:40:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:40:01.883+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:40:03.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:40:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:40:02.796+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:40:03.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:40:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:40:02.796+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:40:03.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:40:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:40:02.796+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:40:03.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:40:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:40:02.797+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:40:03.071 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:40:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:40:02.828+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:40:03.071 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:40:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:40:02.828+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:40:03.071 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:40:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:40:02.828+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:40:03.071 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:40:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:40:02.829+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:40:03.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:40:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:40:02.917+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:40:03.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:40:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:40:02.917+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:40:03.072 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:40:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:40:02.917+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:40:03.072 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:40:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:40:02.918+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:40:03.851 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:40:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:40:03.809+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:40:03.851 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:40:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:40:03.809+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:40:03.851 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:40:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:40:03.809+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:40:03.851 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:40:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:40:03.809+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:40:04.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:40:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:40:03.850+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:40:04.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:40:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:40:03.850+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:40:04.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:40:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:40:03.850+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:40:04.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:40:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:40:03.851+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:40:04.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:40:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:40:03.936+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:40:04.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:40:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:40:03.936+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:40:04.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:40:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:40:03.936+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:40:04.321 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:40:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:40:03.937+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:40:05.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:40:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:40:04.811+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:40:05.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:40:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:40:04.811+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:40:05.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:40:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:40:04.811+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:40:05.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:40:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:40:04.812+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:40:05.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:40:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:40:04.917+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:40:05.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:40:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:40:04.917+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:40:05.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:40:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:40:04.917+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:40:05.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:40:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:40:04.917+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:40:05.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:40:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:40:04.841+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:40:05.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:40:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:40:04.841+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:40:05.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:40:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:40:04.841+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:40:05.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:40:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:40:04.841+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:40:05.791 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:40:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:40:05.370+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 17349 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:40:06.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:40:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:40:05.790+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:40:06.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:40:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:40:05.790+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:40:06.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:40:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:40:05.790+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:40:06.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:40:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:40:05.791+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:40:06.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:40:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:40:05.880+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:40:06.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:40:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:40:05.880+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:40:06.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:40:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:40:05.880+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:40:06.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:40:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:40:05.881+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:40:06.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:40:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:40:05.834+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:40:06.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:40:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:40:05.834+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:40:06.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:40:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:40:05.834+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:40:06.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:40:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:40:05.834+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:40:07.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:40:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:40:06.757+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:40:07.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:40:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:40:06.757+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:40:07.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:40:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:40:06.757+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:40:07.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:40:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:40:06.758+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:40:07.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:40:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:40:06.862+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:40:07.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:40:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:40:06.862+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:40:07.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:40:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:40:06.862+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:40:07.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:40:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:40:06.863+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:40:07.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:40:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:40:06.830+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:40:07.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:40:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:40:06.830+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:40:07.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:40:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:40:06.830+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:40:07.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:40:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:40:06.831+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:40:07.946 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:40:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:40:07.808+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:40:07.946 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:40:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:40:07.808+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:40:07.946 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:40:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:40:07.808+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:40:07.946 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:40:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:40:07.808+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:40:07.947 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:40:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:40:07.731+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:40:07.947 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:40:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:40:07.731+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:40:07.947 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:40:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:40:07.731+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:40:07.947 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:40:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:40:07.732+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:40:07.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:40:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:40:07.816+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:40:07.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:40:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:40:07.816+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:40:07.948 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:40:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:40:07.816+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:40:07.948 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:40:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:40:07.818+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:40:09.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:40:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:40:08.775+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:40:09.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:40:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:40:08.775+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:40:09.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:40:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:40:08.775+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:40:09.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:40:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:40:08.775+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:40:09.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:40:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:40:08.826+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:40:09.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:40:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:40:08.826+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:40:09.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:40:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:40:08.826+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:40:09.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:40:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:40:08.827+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:40:09.071 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:40:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:40:08.763+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:40:09.071 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:40:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:40:08.763+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:40:09.071 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:40:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:40:08.763+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:40:09.071 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:40:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:40:08.763+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:40:10.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:40:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:40:09.869+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:40:10.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:40:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:40:09.869+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:40:10.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:40:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:40:09.869+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:40:10.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:40:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:40:09.870+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:40:10.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:40:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:40:09.790+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:40:10.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:40:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:40:09.790+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:40:10.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:40:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:40:09.790+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:40:10.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:40:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:40:09.791+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:40:10.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:40:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:40:09.817+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:40:10.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:40:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:40:09.817+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:40:10.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:40:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:40:09.817+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:40:10.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:40:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:40:09.818+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:40:10.497 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:40:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:40:10.372+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 17431 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:40:11.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:40:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:40:10.909+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:40:11.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:40:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:40:10.909+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:40:11.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:40:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:40:10.909+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:40:11.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:40:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:40:10.909+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:40:11.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:40:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:40:10.766+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:40:11.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:40:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:40:10.766+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:40:11.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:40:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:40:10.766+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:40:11.071 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:40:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:40:10.768+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:40:11.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:40:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:40:10.781+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:40:11.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:40:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:40:10.781+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:40:11.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:40:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:40:10.781+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:40:11.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:40:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:40:10.781+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:40:12.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:40:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:40:11.910+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:40:12.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:40:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:40:11.910+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:40:12.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:40:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:40:11.910+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:40:12.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:40:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:40:11.910+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:40:12.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:40:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:40:11.811+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:40:12.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:40:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:40:11.811+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:40:12.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:40:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:40:11.811+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:40:12.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:40:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:40:11.811+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:40:12.071 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:40:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:40:11.729+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:40:12.071 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:40:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:40:11.729+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:40:12.071 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:40:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:40:11.729+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:40:12.071 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:40:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:40:11.730+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:40:13.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:40:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:40:12.895+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:40:13.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:40:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:40:12.895+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:40:13.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:40:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:40:12.895+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:40:13.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:40:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:40:12.896+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:40:13.071 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:40:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:40:12.702+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:40:13.071 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:40:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:40:12.702+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:40:13.071 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:40:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:40:12.702+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:40:13.071 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:40:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:40:12.702+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:40:13.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:40:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:40:12.818+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:40:13.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:40:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:40:12.819+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:40:13.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:40:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:40:12.819+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:40:13.072 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:40:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:40:12.819+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:40:13.868 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:40:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:40:13.675+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:40:13.869 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:40:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:40:13.675+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:40:13.869 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:40:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:40:13.675+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:40:13.869 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:40:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:40:13.676+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:40:14.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:40:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:40:13.867+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:40:14.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:40:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:40:13.867+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:40:14.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:40:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:40:13.867+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:40:14.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:40:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:40:13.868+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:40:14.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:40:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:40:13.884+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:40:14.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:40:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:40:13.884+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:40:14.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:40:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:40:13.884+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:40:14.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:40:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:40:13.884+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:40:14.846 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:40:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:40:14.642+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:40:14.847 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:40:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:40:14.642+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:40:14.847 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:40:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:40:14.642+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:40:14.847 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:40:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:40:14.643+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:40:15.322 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:40:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:40:14.895+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:40:15.322 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:40:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:40:14.895+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:40:15.322 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:40:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:40:14.895+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:40:15.323 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:40:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:40:14.896+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:40:15.323 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:40:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:40:14.845+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:40:15.323 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:40:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:40:14.845+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:40:15.323 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:40:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:40:14.845+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:40:15.323 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:40:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:40:14.845+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:40:15.815 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:40:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:40:15.373+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 17513 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:40:15.815 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:40:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:40:15.621+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:40:15.815 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:40:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:40:15.621+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:40:15.815 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:40:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:40:15.621+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:40:15.815 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:40:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:40:15.621+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:40:16.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:40:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:40:15.937+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:40:16.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:40:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:40:15.937+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:40:16.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:40:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:40:15.937+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:40:16.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:40:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:40:15.937+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:40:16.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:40:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:40:15.813+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:40:16.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:40:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:40:15.813+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:40:16.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:40:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:40:15.813+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:40:16.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:40:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:40:15.814+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:40:16.954 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:40:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:40:16.815+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:40:16.958 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:40:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:40:16.815+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:40:16.958 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:40:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:40:16.815+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:40:16.958 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:40:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:40:16.816+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:40:16.959 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:40:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:40:16.573+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:40:16.959 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:40:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:40:16.573+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:40:16.959 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:40:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:40:16.573+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:40:16.959 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:40:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:40:16.574+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:40:17.319 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:40:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:40:16.953+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:40:17.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:40:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:40:16.953+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:40:17.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:40:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:40:16.953+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:40:17.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:40:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:40:16.954+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:40:17.989 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:40:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:40:17.791+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:40:17.990 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:40:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:40:17.791+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:40:17.990 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:40:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:40:17.791+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:40:17.990 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:40:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:40:17.791+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:40:17.990 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:40:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:40:17.601+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:40:17.990 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:40:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:40:17.601+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:40:17.990 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:40:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:40:17.601+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:40:17.991 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:40:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:40:17.602+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:40:18.319 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:40:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:40:17.988+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:40:18.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:40:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:40:17.988+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:40:18.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:40:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:40:17.988+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:40:18.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:40:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:40:17.989+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:40:19.001 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:40:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:40:18.777+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:40:19.002 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:40:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:40:18.777+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:40:19.002 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:40:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:40:18.777+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:40:19.002 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:40:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:40:18.777+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:40:19.002 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:40:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:40:18.590+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:40:19.002 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:40:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:40:18.590+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:40:19.002 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:40:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:40:18.590+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:40:19.003 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:40:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:40:18.591+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:40:19.319 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:40:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:40:19.000+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:40:19.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:40:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:40:19.000+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:40:19.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:40:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:40:19.000+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:40:19.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:40:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:40:19.001+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:40:20.039 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:40:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:40:19.770+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:40:20.039 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:40:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:40:19.770+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:40:20.040 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:40:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:40:19.770+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:40:20.040 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:40:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:40:19.771+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:40:20.040 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:40:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:40:19.603+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:40:20.040 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:40:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:40:19.603+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:40:20.040 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:40:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:40:19.603+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:40:20.040 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:40:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:40:19.604+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:40:20.319 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:40:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:40:20.038+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:40:20.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:40:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:40:20.038+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:40:20.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:40:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:40:20.038+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:40:20.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:40:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:40:20.039+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:40:20.729 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:40:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:40:20.375+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 17595 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:40:20.729 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:40:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:40:20.571+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:40:20.730 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:40:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:40:20.571+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:40:20.730 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:40:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:40:20.571+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:40:20.730 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:40:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:40:20.572+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:40:21.048 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:40:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:40:20.728+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:40:21.049 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:40:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:40:20.728+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:40:21.049 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:40:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:40:20.728+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:40:21.049 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:40:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:40:20.729+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:40:21.319 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:40:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:40:21.047+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:40:21.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:40:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:40:21.047+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:40:21.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:40:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:40:21.047+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:40:21.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:40:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:40:21.048+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:40:21.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:40:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:40:21.554+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:40:21.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:40:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:40:21.554+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:40:21.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:40:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:40:21.554+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:40:21.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:40:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:40:21.555+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:40:21.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:40:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:40:21.692+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:40:21.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:40:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:40:21.692+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:40:21.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:40:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:40:21.692+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:40:21.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:40:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:40:21.692+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:40:22.319 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:40:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:40:22.040+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:40:22.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:40:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:40:22.040+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:40:22.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:40:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:40:22.040+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:40:22.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:40:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:40:22.041+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:40:22.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:40:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:40:22.563+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:40:22.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:40:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:40:22.563+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:40:22.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:40:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:40:22.563+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:40:22.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:40:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:40:22.564+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:40:22.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:40:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:40:22.656+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:40:22.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:40:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:40:22.656+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:40:22.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:40:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:40:22.656+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:40:22.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:40:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:40:22.657+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:40:23.534 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:40:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:40:23.082+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:40:23.534 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:40:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:40:23.082+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:40:23.534 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:40:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:40:23.082+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:40:23.535 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:40:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:40:23.083+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:40:23.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:40:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:40:23.533+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:40:23.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:40:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:40:23.533+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:40:23.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:40:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:40:23.533+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:40:23.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:40:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:40:23.534+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:40:23.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:40:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:40:23.701+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:40:23.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:40:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:40:23.701+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:40:23.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:40:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:40:23.701+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:40:23.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:40:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:40:23.701+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:40:24.319 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:40:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:40:24.075+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:40:24.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:40:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:40:24.075+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:40:24.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:40:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:40:24.075+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:40:24.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:40:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:40:24.076+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:40:24.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:40:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:40:24.549+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:40:24.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:40:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:40:24.549+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:40:24.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:40:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:40:24.549+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:40:24.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:40:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:40:24.550+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:40:24.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:40:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:40:24.742+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:40:24.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:40:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:40:24.742+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:40:24.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:40:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:40:24.742+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:40:24.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:40:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:40:24.742+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:40:25.319 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:40:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:40:25.067+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:40:25.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:40:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:40:25.067+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:40:25.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:40:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:40:25.067+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:40:25.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:40:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:40:25.068+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:40:25.721 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:40:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:40:25.377+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 17677 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:40:25.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:40:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:40:25.545+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:40:25.722 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:40:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:40:25.545+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:40:25.722 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:40:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:40:25.545+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:40:25.722 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:40:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:40:25.546+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:40:26.067 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:40:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:40:25.720+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:40:26.067 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:40:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:40:25.720+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:40:26.067 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:40:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:40:25.720+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:40:26.067 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:40:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:40:25.721+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:40:26.319 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:40:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:40:26.065+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:40:26.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:40:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:40:26.065+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:40:26.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:40:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:40:26.065+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:40:26.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:40:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:40:26.066+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:40:26.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:40:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:40:26.533+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:40:26.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:40:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:40:26.533+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:40:26.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:40:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:40:26.533+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:40:26.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:40:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:40:26.534+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:40:26.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:40:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:40:26.728+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:40:26.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:40:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:40:26.728+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:40:26.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:40:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:40:26.728+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:40:26.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:40:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:40:26.729+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:40:27.319 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:40:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:40:27.053+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:40:27.319 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:40:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:40:27.053+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:40:27.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:40:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:40:27.053+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:40:27.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:40:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:40:27.054+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:40:27.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:40:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:40:27.488+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:40:27.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:40:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:40:27.488+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:40:27.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:40:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:40:27.488+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:40:27.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:40:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:40:27.489+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:40:27.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:40:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:40:27.717+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:40:27.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:40:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:40:27.717+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:40:27.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:40:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:40:27.717+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:40:27.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:40:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:40:27.717+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:40:28.464 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:40:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:40:28.075+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:40:28.464 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:40:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:40:28.075+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:40:28.464 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:40:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:40:28.075+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:40:28.465 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:40:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:40:28.075+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:40:28.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:40:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:40:28.463+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:40:28.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:40:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:40:28.463+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:40:28.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:40:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:40:28.463+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:40:28.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:40:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:40:28.464+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:40:28.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:40:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:40:28.672+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:40:28.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:40:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:40:28.672+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:40:28.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:40:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:40:28.672+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:40:28.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:40:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:40:28.672+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:40:29.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:40:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:40:29.123+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:40:29.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:40:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:40:29.123+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:40:29.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:40:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:40:29.123+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:40:29.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:40:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:40:29.124+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:40:29.446 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:40:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:40:29.444+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:40:29.446 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:40:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:40:29.444+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:40:29.446 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:40:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:40:29.444+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:40:29.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:40:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:40:29.445+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:40:29.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:40:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:40:29.692+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:40:29.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:40:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:40:29.692+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:40:29.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:40:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:40:29.692+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:40:29.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:40:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:40:29.692+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:40:30.380 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:40:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:40:30.114+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:40:30.381 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:40:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:40:30.114+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:40:30.381 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:40:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:40:30.114+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:40:30.381 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:40:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:40:30.115+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:40:30.667 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:40:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:40:30.379+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 17758 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:40:30.668 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:40:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:40:30.479+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:40:30.668 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:40:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:40:30.479+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:40:30.668 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:40:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:40:30.479+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:40:30.668 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:40:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:40:30.480+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:40:31.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:40:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:40:30.666+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:40:31.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:40:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:40:30.666+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:40:31.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:40:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:40:30.666+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:40:31.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:40:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:40:30.667+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:40:31.483 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:40:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:40:31.102+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:40:31.483 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:40:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:40:31.102+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:40:31.483 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:40:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:40:31.102+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:40:31.483 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:40:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:40:31.103+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:40:31.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:40:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:40:31.481+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:40:31.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:40:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:40:31.481+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:40:31.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:40:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:40:31.481+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:40:31.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:40:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:40:31.483+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:40:31.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:40:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:40:31.627+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:40:31.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:40:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:40:31.627+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:40:31.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:40:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:40:31.627+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:40:31.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:40:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:40:31.627+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:40:32.440 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:40:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:40:32.085+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:40:32.441 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:40:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:40:32.085+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:40:32.441 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:40:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:40:32.085+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:40:32.441 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:40:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:40:32.086+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:40:32.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:40:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:40:32.652+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:40:32.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:40:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:40:32.652+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:40:32.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:40:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:40:32.652+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:40:32.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:40:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:40:32.652+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:40:32.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:40:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:40:32.439+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:40:32.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:40:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:40:32.439+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:40:32.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:40:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:40:32.439+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:40:32.821 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:40:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:40:32.440+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:40:33.443 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:40:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:40:33.083+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:40:33.444 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:40:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:40:33.083+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:40:33.444 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:40:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:40:33.083+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:40:33.444 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:40:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:40:33.084+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:40:33.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:40:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:40:33.442+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:40:33.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:40:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:40:33.442+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:40:33.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:40:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:40:33.442+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:40:33.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:40:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:40:33.443+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:40:34.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:40:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:40:33.696+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:40:34.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:40:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:40:33.696+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:40:34.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:40:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:40:33.696+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:40:34.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:40:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:40:33.696+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:40:34.441 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:40:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:40:34.069+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:40:34.441 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:40:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:40:34.069+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:40:34.441 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:40:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:40:34.069+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:40:34.441 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:40:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:40:34.070+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:40:34.735 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:40:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:40:34.440+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:40:34.735 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:40:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:40:34.440+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:40:34.735 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:40:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:40:34.440+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:40:34.736 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:40:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:40:34.441+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:40:35.026 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:40:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:40:34.734+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:40:35.026 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:40:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:40:34.734+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:40:35.027 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:40:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:40:34.734+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:40:35.027 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:40:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:40:34.735+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:40:35.319 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:40:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:40:35.025+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:40:35.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:40:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:40:35.025+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:40:35.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:40:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:40:35.025+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:40:35.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:40:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:40:35.026+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:40:35.730 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:40:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:40:35.381+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 17839 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:40:35.731 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:40:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:40:35.450+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:40:35.731 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:40:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:40:35.450+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:40:35.731 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:40:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:40:35.450+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:40:35.731 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:40:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:40:35.451+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:40:36.067 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:40:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:40:35.729+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:40:36.068 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:40:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:40:35.729+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:40:36.068 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:40:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:40:35.729+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:40:36.068 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:40:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:40:35.730+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:40:36.319 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:40:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:40:36.066+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:40:36.319 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:40:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:40:36.066+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:40:36.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:40:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:40:36.066+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:40:36.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:40:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:40:36.067+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:40:36.705 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:40:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:40:36.437+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:40:36.705 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:40:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:40:36.437+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:40:36.705 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:40:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:40:36.437+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:40:36.705 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:40:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:40:36.437+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:40:37.037 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:40:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:40:36.703+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:40:37.037 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:40:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:40:36.703+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:40:37.037 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:40:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:40:36.703+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:40:37.037 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:40:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:40:36.704+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:40:37.319 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:40:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:40:37.035+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:40:37.319 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:40:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:40:37.035+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:40:37.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:40:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:40:37.035+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:40:37.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:40:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:40:37.036+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:40:37.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:40:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:40:37.396+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:40:37.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:40:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:40:37.396+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:40:37.695 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:40:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:40:37.396+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:40:37.695 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:40:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:40:37.397+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:40:38.045 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:40:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:40:37.693+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:40:38.045 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:40:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:40:37.693+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:40:38.045 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:40:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:40:37.693+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:40:38.045 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:40:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:40:37.693+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:40:38.319 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:40:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:40:38.044+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:40:38.319 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:40:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:40:38.044+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:40:38.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:40:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:40:38.044+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:40:38.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:40:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:40:38.044+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:40:38.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:40:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:40:38.434+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:40:38.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:40:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:40:38.434+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:40:38.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:40:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:40:38.434+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:40:38.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:40:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:40:38.435+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:40:38.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:40:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:40:38.667+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:40:38.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:40:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:40:38.667+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:40:38.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:40:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:40:38.667+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:40:38.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:40:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:40:38.668+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:40:39.417 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:40:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:40:39.088+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:40:39.417 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:40:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:40:39.088+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:40:39.417 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:40:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:40:39.088+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:04.788908+0000 (oldest deadline 2024-03-18T01:30:24.788908+0000) 2024-03-18T01:40:39.417 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:40:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:40:39.089+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:40:39.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:40:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:40:39.415+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:40:39.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:40:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:40:39.415+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:40:39.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:40:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:40:39.415+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:40:39.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:40:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:40:39.416+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:40:39.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:40:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:40:39.663+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:40:39.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:40:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:40:39.663+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:40:39.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:40:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:40:39.663+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:40:39.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:40:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:40:39.663+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:40:40.319 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:40:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:40:40.045+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:40:40.819 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:40:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:40:40.382+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 17921 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:40:40.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:40:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:40:40.621+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:40:40.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:40:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:40:40.621+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:40:40.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:40:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:40:40.621+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:40:40.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:40:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:40:40.622+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:40:40.821 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:40:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:40:40.412+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:40:40.821 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:40:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:40:40.412+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:40:40.821 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:40:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:40:40.412+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:40:40.821 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:40:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:40:40.412+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:40:41.319 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:40:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:40:41.044+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:40:41.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:40:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:40:41.454+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:40:41.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:40:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:40:41.454+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:40:41.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:40:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:40:41.454+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:40:41.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:40:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:40:41.455+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:40:41.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:40:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:40:41.640+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:40:41.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:40:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:40:41.640+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:40:41.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:40:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:40:41.640+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:40:41.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:40:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:40:41.641+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:40:42.319 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:40:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:40:42.046+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:40:42.691 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:40:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:40:42.477+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:40:42.691 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:40:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:40:42.477+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:40:42.691 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:40:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:40:42.477+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:40:42.691 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:40:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:40:42.478+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:40:43.058 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:40:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:40:42.689+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:40:43.059 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:40:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:40:42.689+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:40:43.059 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:40:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:40:42.689+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:40:43.059 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:40:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:40:42.689+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:40:43.319 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:40:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:40:43.056+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:40:43.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:40:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:40:43.502+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:40:43.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:40:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:40:43.502+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:40:43.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:40:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:40:43.502+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:40:43.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:40:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:40:43.502+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:40:43.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:40:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:40:43.721+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:40:43.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:40:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:40:43.721+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:40:43.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:40:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:40:43.721+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:40:43.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:40:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:40:43.721+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:40:44.526 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:40:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:40:44.100+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:40:44.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:40:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:40:44.524+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:40:44.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:40:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:40:44.524+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:40:44.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:40:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:40:44.524+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:40:44.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:40:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:40:44.525+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:40:44.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:40:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:40:44.766+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:40:44.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:40:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:40:44.766+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:40:44.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:40:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:40:44.766+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:40:44.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:40:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:40:44.767+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:40:45.386 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:40:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:40:45.086+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:40:45.767 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:40:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:40:45.384+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 18001 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:40:45.767 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:40:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:40:45.572+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:40:45.767 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:40:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:40:45.572+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:40:45.767 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:40:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:40:45.572+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:40:45.768 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:40:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:40:45.572+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:40:46.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:40:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:40:45.765+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:40:46.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:40:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:40:45.765+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:40:46.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:40:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:40:45.765+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:40:46.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:40:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:40:45.765+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:40:46.539 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:40:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:40:46.132+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:40:46.817 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:40:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:40:46.537+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:40:46.817 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:40:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:40:46.537+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:40:46.818 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:40:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:40:46.537+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:40:46.818 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:40:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:40:46.538+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:40:47.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:40:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:40:46.815+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:40:47.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:40:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:40:46.815+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:40:47.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:40:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:40:46.815+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:40:47.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:40:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:40:46.815+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:40:47.565 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:40:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:40:47.172+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:40:47.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:40:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:40:47.804+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:40:47.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:40:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:40:47.804+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:40:47.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:40:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:40:47.804+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:40:47.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:40:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:40:47.804+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:40:47.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:40:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:40:47.563+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:40:47.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:40:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:40:47.563+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:40:47.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:40:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:40:47.563+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:40:47.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:40:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:40:47.564+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:40:48.529 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:40:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:40:48.178+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:40:48.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:40:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:40:48.527+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:40:48.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:40:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:40:48.527+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:40:48.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:40:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:40:48.527+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:40:48.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:40:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:40:48.528+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:40:49.116 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:40:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:40:48.842+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:40:49.116 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:40:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:40:48.842+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:40:49.116 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:40:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:40:48.842+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:40:49.116 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:40:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:40:48.842+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:40:49.487 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:40:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:40:49.187+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:40:49.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:40:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:40:49.485+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:40:49.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:40:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:40:49.485+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:40:49.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:40:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:40:49.485+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:40:49.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:40:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:40:49.486+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:40:50.171 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:40:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:40:49.883+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:40:50.171 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:40:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:40:49.883+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:40:50.171 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:40:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:40:49.883+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:40:50.171 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:40:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:40:49.883+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:40:50.480 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:40:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:40:50.387+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 18082 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:40:50.481 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:40:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:40:50.185+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:40:50.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:40:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:40:50.479+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:40:50.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:40:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:40:50.479+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:40:50.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:40:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:40:50.479+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:40:50.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:40:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:40:50.480+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:40:51.230 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:40:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:40:50.896+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:40:51.231 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:40:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:40:50.896+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:40:51.231 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:40:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:40:50.896+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:40:51.231 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:40:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:40:50.896+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:40:51.505 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:40:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:40:51.229+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:40:51.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:40:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:40:51.504+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:40:51.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:40:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:40:51.504+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:40:51.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:40:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:40:51.504+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:14.774833+0000 (oldest deadline 2024-03-18T01:30:34.774833+0000) 2024-03-18T01:40:51.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:40:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:40:51.504+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:40:52.190 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:40:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:40:51.928+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:40:52.190 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:40:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:40:51.928+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:40:52.190 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:40:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:40:51.928+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:40:52.190 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:40:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:40:51.929+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:40:52.480 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:40:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:40:52.189+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:40:52.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:40:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:40:52.479+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:40:53.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:40:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:40:52.917+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:40:53.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:40:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:40:52.917+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:40:53.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:40:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:40:52.917+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:30:19.763771+0000 (oldest deadline 2024-03-18T01:30:39.763771+0000) 2024-03-18T01:40:53.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:40:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:40:52.917+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:40:53.469 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:40:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:40:53.194+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:40:53.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:40:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:40:53.468+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:40:54.219 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:40:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:40:53.925+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:40:54.473 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:40:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:40:54.218+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:40:54.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:40:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:40:54.472+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:40:55.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:40:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:40:54.966+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:40:55.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:40:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:40:55.205+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:40:55.819 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:40:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:40:55.388+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 18161 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:40:55.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:40:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:40:55.481+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:40:56.319 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:40:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:40:56.179+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:40:56.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:40:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:40:55.997+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:40:56.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:40:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:40:56.507+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:40:57.319 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:40:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:40:57.165+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:40:57.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:40:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:40:57.022+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:40:57.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:40:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:40:57.513+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:40:58.319 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:40:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:40:58.201+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:40:58.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:40:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:40:57.985+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:40:58.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:40:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:40:58.492+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:40:59.319 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:40:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:40:59.177+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:40:59.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:40:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:40:58.959+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:40:59.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:40:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:40:59.472+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:41:00.203 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:40:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:40:59.951+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:41:00.486 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:41:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:41:00.390+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 18242 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:41:00.487 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:41:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:41:00.202+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:41:00.487 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:41:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:41:00.202+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:41:00.487 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:41:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:41:00.202+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:41:00.487 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:41:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:41:00.202+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:41:00.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:41:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:41:00.485+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:41:01.225 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:41:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:41:00.911+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:41:01.509 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:41:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:41:01.224+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:41:01.509 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:41:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:41:01.224+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:41:01.509 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:41:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:41:01.224+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:41:01.509 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:41:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:41:01.225+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:41:01.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:41:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:41:01.508+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:41:02.192 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:41:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:41:01.891+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:41:02.548 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:41:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:41:02.191+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:41:02.548 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:41:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:41:02.191+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:41:02.549 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:41:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:41:02.191+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:41:02.549 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:41:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:41:02.191+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:41:02.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:41:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:41:02.547+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:41:03.157 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:41:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:41:02.891+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:41:03.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:41:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:41:03.156+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:41:03.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:41:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:41:03.156+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:41:03.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:41:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:41:03.156+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:41:03.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:41:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:41:03.157+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:41:03.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:41:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:41:03.568+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:41:04.171 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:41:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:41:03.881+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:41:04.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:41:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:41:04.171+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:41:04.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:41:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:41:04.171+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:41:04.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:41:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:41:04.171+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:41:04.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:41:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:41:04.171+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:41:04.869 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:41:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:41:04.594+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:41:05.201 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:41:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:41:04.868+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:41:05.517 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:41:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:41:05.392+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 18323 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:41:05.517 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:41:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:41:05.200+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:41:05.517 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:41:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:41:05.200+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:41:05.517 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:41:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:41:05.200+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:41:05.518 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:41:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:41:05.201+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:41:05.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:41:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:41:05.644+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:41:06.177 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:41:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:41:05.833+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:41:06.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:41:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:41:06.176+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:41:06.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:41:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:41:06.176+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:41:06.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:41:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:41:06.176+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:41:06.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:41:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:41:06.177+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:41:07.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:41:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:41:06.689+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:41:07.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:41:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:41:06.877+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:41:07.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:41:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:41:07.138+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:41:07.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:41:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:41:07.138+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:41:07.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:41:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:41:07.138+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:41:07.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:41:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:41:07.139+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:41:08.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:41:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:41:07.716+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:41:08.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:41:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:41:07.919+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:41:08.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:41:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:41:08.112+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:41:08.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:41:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:41:08.112+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:41:08.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:41:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:41:08.112+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:41:08.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:41:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:41:08.113+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:41:09.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:41:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:41:08.717+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:41:09.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:41:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:41:08.875+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:41:09.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:41:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:41:09.104+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:41:09.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:41:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:41:09.104+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:41:09.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:41:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:41:09.104+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:41:09.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:41:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:41:09.105+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:41:10.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:41:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:41:09.677+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:41:10.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:41:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:41:09.888+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:41:10.395 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:41:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:41:10.126+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:41:10.395 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:41:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:41:10.126+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:41:10.395 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:41:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:41:10.126+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:41:10.396 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:41:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:41:10.127+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:41:10.720 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:41:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:41:10.394+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 18404 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:41:11.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:41:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:41:10.719+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:41:11.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:41:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:41:10.904+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:41:11.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:41:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:41:11.151+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:41:11.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:41:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:41:11.151+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:41:11.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:41:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:41:11.151+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:41:11.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:41:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:41:11.151+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:41:11.954 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:41:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:41:11.672+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:41:12.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:41:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:41:11.953+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:41:12.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:41:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:41:12.157+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:41:12.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:41:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:41:12.157+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:41:12.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:41:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:41:12.157+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:41:12.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:41:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:41:12.158+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:41:12.983 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:41:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:41:12.650+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:41:13.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:41:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:41:12.982+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:41:13.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:41:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:41:13.193+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:41:13.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:41:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:41:13.193+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:41:13.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:41:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:41:13.193+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:41:13.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:41:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:41:13.194+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:41:13.992 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:41:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:41:13.687+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:41:14.319 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:41:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:41:14.222+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:41:14.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:41:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:41:14.222+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:41:14.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:41:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:41:14.222+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:41:14.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:41:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:41:14.223+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:41:14.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:41:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:41:13.991+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:41:14.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:41:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:41:13.991+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:41:14.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:41:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:41:13.991+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:41:14.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:41:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:41:13.992+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:41:15.025 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:41:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:41:14.726+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:41:15.026 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:41:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:41:14.727+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:41:15.026 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:41:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:41:14.727+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:41:15.026 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:41:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:41:14.727+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:41:15.319 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:41:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:41:15.238+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:41:15.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:41:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:41:15.238+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:41:15.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:41:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:41:15.238+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:41:15.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:41:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:41:15.239+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:41:15.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:41:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:41:15.024+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:41:15.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:41:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:41:15.024+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:41:15.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:41:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:41:15.024+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:41:15.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:41:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:41:15.025+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:41:15.700 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:41:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:41:15.396+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 18485 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:41:16.041 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:41:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:41:15.699+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:41:16.041 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:41:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:41:15.699+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:41:16.041 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:41:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:41:15.699+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:41:16.041 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:41:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:41:15.700+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:41:16.319 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:41:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:41:16.249+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:41:16.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:41:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:41:16.249+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:41:16.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:41:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:41:16.249+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:41:16.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:41:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:41:16.250+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:41:16.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:41:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:41:16.039+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:41:16.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:41:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:41:16.039+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:41:16.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:41:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:41:16.039+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:41:16.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:41:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:41:16.040+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:41:17.010 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:41:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:41:16.722+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:41:17.011 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:41:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:41:16.722+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:41:17.011 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:41:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:41:16.722+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:41:17.011 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:41:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:41:16.723+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:41:17.283 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:41:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:41:17.009+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:41:17.283 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:41:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:41:17.009+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:41:17.283 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:41:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:41:17.009+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:41:17.284 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:41:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:41:17.010+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:41:17.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:41:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:41:17.282+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:41:17.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:41:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:41:17.282+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:41:17.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:41:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:41:17.282+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:41:17.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:41:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:41:17.283+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:41:18.044 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:41:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:41:17.712+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:41:18.044 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:41:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:41:17.712+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:41:18.044 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:41:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:41:17.712+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:41:18.044 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:41:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:41:17.712+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:41:18.319 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:41:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:41:18.289+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:41:18.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:41:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:41:18.289+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:41:18.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:41:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:41:18.289+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:41:18.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:41:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:41:18.289+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:41:18.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:41:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:41:18.043+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:41:18.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:41:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:41:18.043+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:41:18.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:41:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:41:18.043+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:41:18.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:41:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:41:18.043+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:41:19.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:41:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:41:18.762+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:41:19.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:41:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:41:18.762+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:41:19.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:41:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:41:18.762+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:41:19.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:41:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:41:18.762+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:41:19.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:41:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:41:19.321+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:41:19.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:41:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:41:19.321+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:41:19.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:41:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:41:19.321+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:41:19.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:41:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:41:19.322+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:41:19.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:41:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:41:19.088+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:41:19.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:41:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:41:19.088+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:41:19.571 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:41:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:41:19.088+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:41:19.571 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:41:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:41:19.088+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:41:20.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:41:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:41:19.810+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:41:20.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:41:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:41:19.810+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:41:20.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:41:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:41:19.810+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:41:20.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:41:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:41:19.811+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:41:20.399 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:41:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:41:20.307+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:41:20.399 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:41:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:41:20.307+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:41:20.399 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:41:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:41:20.307+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:41:20.399 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:41:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:41:20.308+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:41:20.399 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:41:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:41:20.115+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:41:20.400 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:41:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:41:20.116+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:41:20.400 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:41:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:41:20.116+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:41:20.400 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:41:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:41:20.116+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:41:20.789 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:41:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:41:20.398+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 18566 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:41:21.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:41:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:41:20.787+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:41:21.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:41:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:41:20.787+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:41:21.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:41:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:41:20.787+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:41:21.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:41:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:41:20.788+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:41:21.341 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:41:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:41:21.072+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:41:21.342 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:41:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:41:21.072+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:41:21.342 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:41:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:41:21.072+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:41:21.342 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:41:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:41:21.073+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:41:21.704 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:41:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:41:21.340+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:41:21.705 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:41:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:41:21.340+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:41:21.705 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:41:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:41:21.340+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:41:21.705 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:41:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:41:21.341+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:41:22.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:41:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:41:21.753+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:41:22.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:41:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:41:21.753+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:41:22.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:41:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:41:21.753+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:41:22.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:41:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:41:21.754+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:41:22.390 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:41:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:41:22.095+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:41:22.390 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:41:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:41:22.095+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:41:22.390 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:41:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:41:22.095+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:41:22.390 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:41:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:41:22.095+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:41:22.801 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:41:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:41:22.389+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:41:22.801 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:41:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:41:22.389+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:41:22.801 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:41:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:41:22.389+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:41:22.801 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:41:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:41:22.390+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:41:23.065 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:41:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:41:22.800+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:41:23.065 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:41:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:41:22.800+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:41:23.065 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:41:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:41:22.800+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:41:23.065 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:41:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:41:22.801+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:41:23.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:41:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:41:23.063+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:41:23.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:41:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:41:23.063+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:41:23.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:41:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:41:23.063+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:41:23.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:41:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:41:23.064+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:41:23.784 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:41:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:41:23.402+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:41:23.785 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:41:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:41:23.402+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:41:23.785 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:41:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:41:23.402+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:41:23.785 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:41:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:41:23.403+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:41:24.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:41:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:41:23.783+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:41:24.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:41:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:41:23.783+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:41:24.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:41:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:41:23.783+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:41:24.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:41:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:41:23.784+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:41:24.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:41:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:41:24.028+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:41:24.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:41:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:41:24.028+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:41:24.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:41:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:41:24.028+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:41:24.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:41:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:41:24.028+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:41:24.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:41:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:41:24.364+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:41:24.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:41:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:41:24.364+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:41:24.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:41:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:41:24.364+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:41:24.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:41:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:41:24.365+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:41:25.321 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:41:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:41:24.831+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:41:25.322 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:41:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:41:24.831+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:41:25.322 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:41:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:41:24.831+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:41:25.322 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:41:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:41:24.832+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:41:25.322 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:41:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:41:25.037+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:41:25.323 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:41:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:41:25.037+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:41:25.323 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:41:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:41:25.037+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:41:25.323 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:41:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:41:25.037+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:41:25.819 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:41:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:41:25.399+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 18646 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:41:25.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:41:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:41:25.342+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:41:25.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:41:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:41:25.342+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:41:25.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:41:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:41:25.342+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:41:25.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:41:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:41:25.342+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:41:26.308 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:41:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:41:25.858+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:41:26.309 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:41:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:41:25.858+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:41:26.309 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:41:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:41:25.858+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:41:26.309 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:41:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:41:25.858+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:41:26.309 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:41:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:41:26.025+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:41:26.309 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:41:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:41:26.025+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:41:26.309 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:41:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:41:26.025+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:41:26.310 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:41:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:41:26.025+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:41:26.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:41:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:41:26.307+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:41:26.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:41:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:41:26.307+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:41:26.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:41:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:41:26.307+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:41:26.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:41:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:41:26.308+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:41:27.300 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:41:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:41:27.024+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:41:27.300 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:41:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:41:27.024+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:41:27.300 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:41:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:41:27.024+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:41:27.300 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:41:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:41:27.025+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:41:27.301 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:41:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:41:26.867+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:41:27.301 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:41:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:41:26.867+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:41:27.301 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:41:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:41:26.867+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:41:27.301 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:41:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:41:26.868+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:41:27.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:41:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:41:27.298+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:41:27.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:41:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:41:27.298+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:41:27.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:41:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:41:27.298+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:41:27.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:41:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:41:27.299+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:41:28.289 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:41:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:41:27.994+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:41:28.289 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:41:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:41:27.994+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:41:28.289 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:41:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:41:27.994+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:41:28.289 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:41:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:41:27.994+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:41:28.289 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:41:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:41:27.869+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:41:28.290 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:41:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:41:27.869+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:41:28.290 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:41:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:41:27.869+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:41:28.290 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:41:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:41:27.870+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:41:28.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:41:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:41:28.287+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:41:28.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:41:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:41:28.287+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:41:28.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:41:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:41:28.287+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:41:28.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:41:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:41:28.288+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:41:29.271 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:41:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:41:28.909+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:41:29.271 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:41:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:41:28.909+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:41:29.271 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:41:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:41:28.909+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:41:29.271 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:41:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:41:28.909+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:41:29.272 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:41:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:41:29.030+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:41:29.272 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:41:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:41:29.030+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:41:29.272 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:41:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:41:29.030+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:41:29.272 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:41:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:41:29.030+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:41:29.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:41:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:41:29.270+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:41:29.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:41:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:41:29.270+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:41:29.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:41:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:41:29.270+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:41:29.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:41:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:41:29.270+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:41:30.173 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:41:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:41:30.040+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:41:30.173 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:41:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:41:30.040+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:41:30.173 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:41:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:41:30.040+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:41:30.173 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:41:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:41:30.040+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:41:30.174 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:41:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:41:29.865+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:41:30.174 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:41:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:41:29.865+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:41:30.174 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:41:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:41:29.865+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:41:30.174 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:41:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:41:29.866+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:41:30.427 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:41:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:41:30.401+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 18726 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:41:30.427 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:41:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:41:30.294+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:41:30.427 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:41:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:41:30.294+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:41:30.427 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:41:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:41:30.294+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:41:30.428 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:41:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:41:30.295+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:41:31.274 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:41:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:41:31.085+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:41:31.275 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:41:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:41:31.085+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:41:31.275 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:41:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:41:31.085+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:41:31.275 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:41:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:41:31.085+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:41:31.275 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:41:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:41:30.880+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:41:31.275 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:41:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:41:30.880+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:41:31.275 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:41:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:41:30.880+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:41:31.276 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:41:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:41:30.881+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:41:31.547 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:41:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:41:31.273+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:41:31.547 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:41:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:41:31.273+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:41:31.547 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:41:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:41:31.273+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:41:31.547 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:41:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:41:31.274+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:41:32.122 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:41:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:41:31.852+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:41:32.122 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:41:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:41:31.852+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:41:32.122 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:41:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:41:31.852+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:41:32.122 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:41:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:41:31.853+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:41:32.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:41:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:41:32.259+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:41:32.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:41:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:41:32.259+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:41:32.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:41:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:41:32.259+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:41:32.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:41:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:41:32.259+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:41:32.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:41:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:41:32.120+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:41:32.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:41:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:41:32.120+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:41:32.571 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:41:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:41:32.120+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:41:32.571 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:41:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:41:32.121+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:41:33.231 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:41:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:41:33.120+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:41:33.231 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:41:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:41:33.120+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:41:33.231 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:41:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:41:33.120+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:41:33.232 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:41:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:41:33.120+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:41:33.232 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:41:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:41:32.879+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:41:33.232 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:41:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:41:32.879+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:41:33.232 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:41:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:41:32.879+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:41:33.232 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:41:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:41:32.880+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:41:33.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:41:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:41:33.230+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:41:33.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:41:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:41:33.230+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:41:33.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:41:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:41:33.230+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:41:33.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:41:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:41:33.230+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:41:34.267 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:41:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:41:33.893+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:41:34.268 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:41:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:41:33.893+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:41:34.268 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:41:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:41:33.893+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:41:34.268 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:41:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:41:33.894+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:41:34.268 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:41:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:41:34.076+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:41:34.268 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:41:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:41:34.076+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:41:34.268 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:41:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:41:34.076+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:41:34.269 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:41:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:41:34.077+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:41:34.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:41:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:41:34.266+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:41:34.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:41:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:41:34.266+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:41:34.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:41:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:41:34.266+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:41:34.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:41:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:41:34.267+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:41:35.038 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:41:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:41:34.861+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:41:35.038 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:41:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:41:34.861+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:41:35.038 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:41:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:41:34.861+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:41:35.038 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:41:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:41:34.862+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:41:35.296 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:41:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:41:35.037+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:41:35.296 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:41:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:41:35.037+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:41:35.296 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:41:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:41:35.037+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:41:35.296 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:41:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:41:35.037+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:41:35.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:41:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:41:35.403+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 18806 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:41:35.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:41:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:41:35.294+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:41:35.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:41:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:41:35.294+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:41:35.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:41:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:41:35.294+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:41:35.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:41:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:41:35.295+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:41:36.286 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:41:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:41:36.010+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:41:36.287 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:41:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:41:36.010+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:41:36.287 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:41:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:41:36.010+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:41:36.287 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:41:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:41:36.010+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:41:36.287 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:41:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:41:35.850+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:41:36.287 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:41:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:41:35.850+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:41:36.288 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:41:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:41:35.850+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:41:36.288 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:41:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:41:35.851+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:41:36.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:41:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:41:36.285+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:41:36.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:41:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:41:36.285+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:41:36.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:41:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:41:36.285+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:41:36.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:41:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:41:36.286+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:41:37.270 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:41:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:41:37.056+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:41:37.270 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:41:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:41:37.056+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:41:37.271 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:41:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:41:37.056+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:41:37.271 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:41:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:41:37.057+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:41:37.271 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:41:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:41:36.822+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:41:37.271 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:41:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:41:36.822+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:41:37.271 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:41:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:41:36.822+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:41:37.271 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:41:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:41:36.823+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:41:37.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:41:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:41:37.269+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:41:37.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:41:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:41:37.269+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:41:37.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:41:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:41:37.269+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:41:37.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:41:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:41:37.270+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:41:38.254 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:41:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:41:38.053+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:41:38.254 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:41:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:41:38.053+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:41:38.254 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:41:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:41:38.053+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:41:38.254 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:41:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:41:38.054+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:41:38.255 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:41:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:41:37.853+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:41:38.255 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:41:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:41:37.853+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:41:38.255 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:41:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:41:37.853+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:41:38.255 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:41:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:41:37.854+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:41:38.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:41:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:41:38.253+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:41:38.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:41:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:41:38.253+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:41:38.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:41:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:41:38.253+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:41:38.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:41:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:41:38.253+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:41:39.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:41:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:41:39.042+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:41:39.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:41:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:41:39.042+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:41:39.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:41:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:41:39.042+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:41:39.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:41:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:41:39.042+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:41:39.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:41:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:41:38.814+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:41:39.071 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:41:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:41:38.814+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:41:39.071 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:41:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:41:38.814+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:41:39.071 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:41:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:41:38.816+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:41:39.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:41:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:41:39.302+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:41:39.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:41:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:41:39.302+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:41:39.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:41:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:41:39.302+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:41:39.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:41:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:41:39.303+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:41:40.174 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:41:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:41:40.082+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:41:40.174 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:41:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:41:40.082+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:41:40.174 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:41:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:41:40.082+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:41:40.174 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:41:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:41:40.083+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:41:40.175 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:41:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:41:39.839+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:41:40.175 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:41:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:41:39.839+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:41:40.175 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:41:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:41:39.839+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:41:40.175 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:41:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:41:39.839+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:41:40.427 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:41:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:41:40.404+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 18885 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:41:40.427 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:41:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:41:40.300+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:41:40.428 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:41:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:41:40.300+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:41:40.428 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:41:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:41:40.300+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:41:40.428 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:41:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:41:40.301+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:41:41.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:41:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:41:40.854+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:41:41.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:41:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:41:40.854+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:41:41.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:41:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:41:40.854+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:41:41.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:41:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:41:40.855+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:41:41.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:41:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:41:41.271+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:41:41.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:41:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:41:41.271+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:41:41.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:41:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:41:41.271+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:41:41.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:41:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:41:41.271+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:41:41.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:41:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:41:41.072+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:41:41.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:41:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:41:41.072+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:41:41.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:41:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:41:41.072+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:41:41.571 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:41:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:41:41.073+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:41:42.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:41:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:41:41.855+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:41:42.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:41:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:41:41.855+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:41:42.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:41:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:41:41.855+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:41:42.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:41:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:41:41.855+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:41:42.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:41:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:41:42.277+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:41:42.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:41:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:41:42.277+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:41:42.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:41:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:41:42.277+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:41:42.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:41:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:41:42.278+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:41:42.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:41:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:41:42.078+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:41:42.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:41:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:41:42.078+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:41:42.571 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:41:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:41:42.078+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:41:42.571 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:41:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:41:42.079+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:41:43.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:41:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:41:43.060+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:41:43.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:41:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:41:43.060+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:41:43.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:41:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:41:43.060+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:41:43.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:41:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:41:43.061+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:41:43.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:41:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:41:42.865+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:41:43.321 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:41:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:41:42.865+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:41:43.321 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:41:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:41:42.865+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:41:43.321 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:41:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:41:42.866+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:41:43.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:41:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:41:43.327+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:41:43.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:41:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:41:43.327+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:41:43.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:41:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:41:43.327+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:41:43.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:41:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:41:43.328+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:41:44.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:41:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:41:43.897+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:41:44.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:41:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:41:43.897+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:41:44.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:41:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:41:43.897+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:41:44.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:41:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:41:43.897+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:41:44.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:41:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:41:44.073+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:41:44.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:41:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:41:44.073+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:41:44.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:41:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:41:44.073+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:41:44.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:41:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:41:44.073+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:41:44.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:41:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:41:44.360+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:41:44.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:41:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:41:44.360+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:41:44.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:41:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:41:44.360+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:41:44.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:41:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:41:44.361+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:41:45.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:41:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:41:44.909+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:41:45.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:41:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:41:44.909+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:41:45.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:41:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:41:44.909+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:41:45.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:41:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:41:44.910+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:41:45.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:41:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:41:45.059+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:41:45.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:41:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:41:45.059+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:41:45.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:41:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:41:45.059+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:41:45.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:41:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:41:45.060+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:41:45.819 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:41:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:41:45.406+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 18969 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:41:45.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:41:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:41:45.358+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:41:45.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:41:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:41:45.358+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:41:45.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:41:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:41:45.358+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:41:45.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:41:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:41:45.358+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:41:46.316 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:41:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:41:45.866+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:41:46.316 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:41:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:41:45.866+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:41:46.316 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:41:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:41:45.866+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:41:46.316 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:41:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:41:45.866+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:41:46.316 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:41:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:41:46.054+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:41:46.317 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:41:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:41:46.054+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:41:46.317 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:41:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:41:46.054+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:41:46.317 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:41:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:41:46.054+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:41:46.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:41:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:41:46.314+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:41:46.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:41:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:41:46.314+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:41:46.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:41:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:41:46.314+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:41:46.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:41:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:41:46.314+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:41:47.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:41:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:41:47.034+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:41:47.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:41:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:41:47.034+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:41:47.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:41:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:41:47.034+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:41:47.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:41:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:41:47.034+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:41:47.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:41:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:41:46.857+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:41:47.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:41:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:41:46.857+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:41:47.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:41:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:41:46.857+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:41:47.321 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:41:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:41:46.858+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:41:47.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:41:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:41:47.359+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:41:47.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:41:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:41:47.359+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:41:47.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:41:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:41:47.359+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:41:47.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:41:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:41:47.360+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:41:48.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:41:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:41:47.831+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:41:48.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:41:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:41:47.831+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:41:48.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:41:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:41:47.831+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:41:48.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:41:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:41:47.832+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:41:48.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:41:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:41:48.006+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:41:48.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:41:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:41:48.006+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:41:48.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:41:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:41:48.006+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:41:48.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:41:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:41:48.006+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:41:48.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:41:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:41:48.343+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:41:48.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:41:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:41:48.343+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:41:48.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:41:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:41:48.343+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:41:48.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:41:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:41:48.343+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:41:49.115 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:41:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:41:48.974+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:41:49.116 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:41:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:41:48.974+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:41:49.116 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:41:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:41:48.974+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:41:49.116 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:41:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:41:48.975+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:41:49.116 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:41:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:41:48.831+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:41:49.116 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:41:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:41:48.831+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:41:49.117 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:41:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:41:48.831+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:41:49.117 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:41:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:41:48.832+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:41:49.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:41:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:41:49.300+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:41:49.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:41:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:41:49.300+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:41:49.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:41:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:41:49.300+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:41:49.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:41:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:41:49.301+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:41:50.174 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:41:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:41:49.851+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:41:50.174 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:41:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:41:49.851+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:41:50.174 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:41:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:41:49.851+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:41:50.175 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:41:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:41:49.851+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:41:50.175 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:41:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:41:50.013+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:41:50.175 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:41:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:41:50.013+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:41:50.175 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:41:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:41:50.013+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:41:50.175 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:41:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:41:50.013+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:41:50.430 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:41:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:41:50.409+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 19051 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:41:50.431 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:41:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:41:50.300+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:41:50.431 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:41:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:41:50.300+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:41:50.431 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:41:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:41:50.300+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:41:50.431 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:41:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:41:50.301+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:41:51.263 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:41:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:41:51.056+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:41:51.263 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:41:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:41:51.056+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:41:51.264 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:41:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:41:51.056+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:41:51.264 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:41:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:41:51.056+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:41:51.264 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:41:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:41:50.900+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:41:51.264 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:41:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:41:50.900+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:41:51.264 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:41:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:41:50.900+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:41:51.264 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:41:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:41:50.901+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:41:51.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:41:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:41:51.262+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:41:51.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:41:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:41:51.262+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:41:51.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:41:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:41:51.262+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:41:51.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:41:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:41:51.263+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:41:52.013 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:41:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:41:51.863+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:41:52.014 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:41:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:41:51.863+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:41:52.014 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:41:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:41:51.863+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:41:52.014 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:41:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:41:51.864+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:41:52.267 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:41:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:41:52.012+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:41:52.267 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:41:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:41:52.012+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:41:52.267 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:41:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:41:52.012+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:41:52.267 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:41:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:41:52.013+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:41:52.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:41:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:41:52.265+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:41:52.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:41:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:41:52.266+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:41:52.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:41:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:41:52.266+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:41:52.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:41:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:41:52.266+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:41:53.237 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:41:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:41:53.015+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:41:53.237 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:41:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:41:53.015+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:41:53.238 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:41:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:41:53.015+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:41:53.238 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:41:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:41:53.015+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:41:53.238 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:41:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:41:52.908+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:41:53.238 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:41:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:41:52.908+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:41:53.238 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:41:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:41:52.908+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:41:53.238 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:41:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:41:52.909+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:41:53.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:41:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:41:53.236+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:41:53.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:41:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:41:53.236+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:41:53.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:41:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:41:53.236+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:41:53.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:41:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:41:53.237+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:41:54.203 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:41:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:41:53.863+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:41:54.204 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:41:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:41:53.863+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:41:54.204 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:41:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:41:53.863+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:41:54.204 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:41:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:41:53.864+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:41:54.204 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:41:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:41:54.047+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:41:54.204 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:41:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:41:54.047+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:41:54.204 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:41:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:41:54.047+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:41:54.204 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:41:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:41:54.048+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:41:54.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:41:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:41:54.202+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:41:54.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:41:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:41:54.202+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:41:54.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:41:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:41:54.202+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:41:54.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:41:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:41:54.203+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:41:55.236 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:41:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:41:54.842+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:41:55.237 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:41:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:41:54.842+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:41:55.237 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:41:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:41:54.842+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:41:55.237 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:41:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:41:54.843+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:41:55.237 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:41:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:41:55.087+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:41:55.237 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:41:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:41:55.087+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:41:55.237 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:41:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:41:55.087+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:41:55.238 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:41:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:41:55.087+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:41:55.535 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:41:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:41:55.411+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 19135 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:41:55.535 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:41:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:41:55.235+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:41:55.535 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:41:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:41:55.235+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:41:55.535 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:41:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:41:55.235+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:41:55.535 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:41:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:41:55.236+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:41:56.235 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:41:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:41:55.885+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:41:56.236 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:41:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:41:55.885+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:41:56.236 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:41:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:41:55.885+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:41:56.236 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:41:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:41:55.886+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:41:56.236 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:41:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:41:56.122+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:41:56.236 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:41:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:41:56.122+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:41:56.237 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:41:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:41:56.123+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:41:56.237 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:41:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:41:56.123+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:41:56.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:41:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:41:56.234+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:41:56.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:41:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:41:56.234+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:41:56.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:41:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:41:56.234+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:41:56.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:41:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:41:56.235+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:41:57.141 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:41:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:41:56.851+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:41:57.141 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:41:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:41:56.851+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:41:57.142 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:41:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:41:56.851+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:41:57.142 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:41:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:41:56.852+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:41:57.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:41:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:41:57.271+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:41:57.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:41:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:41:57.271+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:41:57.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:41:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:41:57.271+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:41:57.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:41:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:41:57.272+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:41:57.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:41:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:41:57.140+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:41:57.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:41:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:41:57.140+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:41:57.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:41:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:41:57.140+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:41:57.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:41:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:41:57.140+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:41:58.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:41:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:41:57.806+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:41:58.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:41:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:41:57.806+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:41:58.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:41:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:41:57.806+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:41:58.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:41:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:41:57.806+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:41:58.569 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:41:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:41:58.145+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:41:58.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:41:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:41:58.145+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:41:58.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:41:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:41:58.145+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:41:58.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:41:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:41:58.145+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:41:58.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:41:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:41:58.244+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:41:58.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:41:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:41:58.244+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:41:58.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:41:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:41:58.244+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:41:58.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:41:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:41:58.244+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:41:59.101 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:41:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:41:58.825+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:41:59.102 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:41:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:41:58.825+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:41:59.102 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:41:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:41:58.825+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:41:59.102 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:41:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:41:58.826+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:41:59.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:41:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:41:59.240+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:41:59.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:41:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:41:59.240+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:41:59.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:41:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:41:59.240+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:41:59.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:41:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:41:59.241+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:41:59.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:41:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:41:59.100+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:41:59.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:41:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:41:59.100+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:41:59.571 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:41:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:41:59.100+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:41:59.571 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:41:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:41:59.101+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:42:00.175 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:41:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:41:59.828+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:42:00.176 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:41:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:41:59.828+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:42:00.176 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:41:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:41:59.828+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:42:00.176 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:41:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:41:59.829+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:42:00.176 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:42:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:42:00.057+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:42:00.176 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:42:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:42:00.057+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:42:00.177 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:42:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:42:00.057+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:42:00.177 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:42:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:42:00.057+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:42:00.434 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:42:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:42:00.413+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 19217 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:42:00.435 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:42:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:42:00.242+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:42:00.435 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:42:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:42:00.242+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:42:00.435 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:42:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:42:00.242+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:42:00.435 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:42:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:42:00.243+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:42:01.260 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:42:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:42:01.015+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:42:01.260 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:42:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:42:01.015+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:42:01.260 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:42:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:42:01.015+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:42:01.260 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:42:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:42:01.015+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:42:01.260 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:42:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:42:00.841+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:42:01.261 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:42:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:42:00.841+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:42:01.261 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:42:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:42:00.841+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:42:01.261 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:42:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:42:00.842+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:42:01.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:42:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:42:01.259+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:42:01.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:42:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:42:01.259+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:42:01.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:42:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:42:01.259+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:42:01.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:42:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:42:01.259+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:42:02.062 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:42:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:42:01.841+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:42:02.063 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:42:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:42:01.841+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:42:02.063 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:42:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:42:01.841+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:42:02.063 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:42:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:42:01.842+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:42:02.319 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:42:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:42:02.304+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:42:02.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:42:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:42:02.304+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:42:02.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:42:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:42:02.304+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:42:02.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:42:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:42:02.305+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:42:02.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:42:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:42:02.061+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:42:02.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:42:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:42:02.061+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:42:02.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:42:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:42:02.061+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:42:02.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:42:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:42:02.062+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:42:03.262 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:42:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:42:02.892+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:42:03.263 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:42:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:42:02.892+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:42:03.263 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:42:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:42:02.892+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:42:03.263 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:42:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:42:02.893+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:42:03.263 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:42:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:42:03.094+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:42:03.263 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:42:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:42:03.094+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:42:03.263 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:42:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:42:03.094+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:42:03.264 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:42:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:42:03.094+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:42:03.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:42:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:42:03.261+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:42:03.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:42:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:42:03.261+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:42:03.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:42:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:42:03.261+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:42:03.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:42:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:42:03.262+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:42:04.312 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:42:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:42:03.853+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:42:04.313 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:42:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:42:03.853+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:42:04.313 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:42:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:42:03.853+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:42:04.313 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:42:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:42:03.854+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:42:04.313 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:42:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:42:04.045+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:42:04.313 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:42:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:42:04.045+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:42:04.313 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:42:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:42:04.045+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:42:04.313 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:42:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:42:04.046+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:42:04.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:42:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:42:04.311+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:42:04.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:42:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:42:04.311+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:42:04.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:42:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:42:04.311+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:42:04.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:42:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:42:04.312+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:42:05.079 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:42:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:42:04.826+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:42:05.080 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:42:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:42:04.826+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:42:05.080 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:42:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:42:04.826+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:42:05.080 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:42:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:42:04.827+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:42:05.416 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:42:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:42:05.294+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:42:05.417 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:42:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:42:05.294+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:42:05.417 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:42:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:42:05.294+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:42:05.417 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:42:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:42:05.294+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:42:05.417 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:42:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:42:05.078+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:42:05.417 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:42:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:42:05.078+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:42:05.418 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:42:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:42:05.078+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:42:05.418 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:42:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:42:05.079+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:42:05.819 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:42:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:42:05.415+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 19301 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:42:06.306 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:42:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:42:05.854+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:42:06.306 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:42:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:42:05.854+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:42:06.306 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:42:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:42:05.854+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:42:06.307 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:42:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:42:05.855+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:42:06.307 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:42:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:42:06.046+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:42:06.307 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:42:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:42:06.046+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:42:06.307 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:42:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:42:06.046+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:42:06.307 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:42:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:42:06.047+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:42:06.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:42:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:42:06.305+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:42:06.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:42:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:42:06.305+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:42:06.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:42:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:42:06.305+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:42:06.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:42:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:42:06.306+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:42:07.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:42:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:42:07.008+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:42:07.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:42:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:42:07.008+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:42:07.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:42:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:42:07.008+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:42:07.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:42:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:42:07.009+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:42:07.071 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:42:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:42:06.816+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:42:07.071 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:42:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:42:06.816+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:42:07.071 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:42:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:42:06.816+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:42:07.071 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:42:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:42:06.816+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:42:07.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:42:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:42:07.305+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:42:07.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:42:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:42:07.305+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:42:07.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:42:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:42:07.305+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:42:07.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:42:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:42:07.305+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:42:08.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:42:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:42:07.853+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:42:08.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:42:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:42:07.853+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:42:08.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:42:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:42:07.853+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:42:08.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:42:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:42:07.854+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:42:08.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:42:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:42:07.962+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:42:08.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:42:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:42:07.962+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:42:08.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:42:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:42:07.962+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:42:08.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:42:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:42:07.963+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:42:08.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:42:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:42:08.348+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:42:08.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:42:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:42:08.348+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:42:08.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:42:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:42:08.348+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:42:08.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:42:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:42:08.349+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:42:09.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:42:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:42:08.893+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:42:09.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:42:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:42:08.893+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:42:09.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:42:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:42:08.893+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:42:09.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:42:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:42:08.894+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:42:09.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:42:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:42:08.969+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:42:09.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:42:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:42:08.969+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:42:09.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:42:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:42:08.969+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:42:09.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:42:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:42:08.969+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:42:09.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:42:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:42:09.354+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:42:09.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:42:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:42:09.354+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:42:09.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:42:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:42:09.354+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:42:09.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:42:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:42:09.355+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:42:10.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:42:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:42:09.927+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:42:10.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:42:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:42:09.927+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:42:10.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:42:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:42:09.927+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:42:10.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:42:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:42:09.927+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:42:10.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:42:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:42:09.934+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:42:10.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:42:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:42:09.934+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:42:10.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:42:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:42:09.934+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:42:10.321 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:42:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:42:09.935+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:42:10.820 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:42:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:42:10.417+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 19383 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:42:10.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:42:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:42:10.340+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:42:10.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:42:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:42:10.340+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:42:10.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:42:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:42:10.340+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:42:10.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:42:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:42:10.340+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:42:11.314 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:42:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:42:10.939+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:42:11.314 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:42:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:42:10.939+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:42:11.315 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:42:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:42:10.939+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:42:11.315 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:42:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:42:10.939+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:42:11.315 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:42:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:42:10.917+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:42:11.315 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:42:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:42:10.917+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:42:11.315 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:42:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:42:10.917+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:42:11.315 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:42:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:42:10.918+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:42:11.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:42:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:42:11.313+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:42:11.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:42:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:42:11.313+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:42:11.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:42:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:42:11.313+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:42:11.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:42:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:42:11.314+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:42:12.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:42:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:42:11.942+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:42:12.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:42:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:42:11.942+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:42:12.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:42:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:42:11.942+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:42:12.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:42:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:42:11.943+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:42:12.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:42:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:42:11.915+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:42:12.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:42:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:42:11.915+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:42:12.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:42:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:42:11.915+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:42:12.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:42:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:42:11.915+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:42:12.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:42:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:42:12.346+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:42:12.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:42:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:42:12.346+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:42:12.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:42:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:42:12.346+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:42:12.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:42:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:42:12.347+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:42:13.306 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:42:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:42:12.955+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:42:13.306 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:42:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:42:12.955+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:42:13.307 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:42:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:42:12.955+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:42:13.307 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:42:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:42:12.956+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:42:13.307 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:42:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:42:12.954+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:42:13.307 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:42:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:42:12.954+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:42:13.307 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:42:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:42:12.954+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:42:13.307 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:42:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:42:12.954+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:42:13.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:42:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:42:13.305+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:42:13.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:42:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:42:13.305+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:42:13.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:42:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:42:13.305+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:42:13.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:42:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:42:13.306+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:42:14.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:42:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:42:13.928+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:42:14.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:42:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:42:13.928+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:42:14.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:42:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:42:13.928+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:42:14.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:42:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:42:13.928+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:42:14.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:42:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:42:13.936+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:42:14.321 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:42:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:42:13.936+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:42:14.321 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:42:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:42:13.936+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:42:14.321 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:42:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:42:13.937+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:42:14.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:42:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:42:14.329+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:42:14.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:42:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:42:14.329+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:42:14.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:42:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:42:14.329+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:42:14.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:42:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:42:14.330+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:42:15.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:42:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:42:14.920+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:42:15.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:42:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:42:14.920+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:42:15.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:42:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:42:14.920+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:42:15.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:42:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:42:14.921+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:42:15.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:42:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:42:14.959+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:42:15.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:42:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:42:14.959+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:42:15.321 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:42:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:42:14.959+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:42:15.321 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:42:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:42:14.960+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:42:15.820 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:42:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:42:15.419+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 19467 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:42:15.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:42:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:42:15.323+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:42:15.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:42:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:42:15.323+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:42:15.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:42:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:42:15.323+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:42:15.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:42:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:42:15.324+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:42:16.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:42:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:42:15.922+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:42:16.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:42:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:42:15.922+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:42:16.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:42:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:42:15.922+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:42:16.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:42:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:42:15.922+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:42:16.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:42:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:42:16.008+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:42:16.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:42:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:42:16.008+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:42:16.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:42:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:42:16.008+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:42:16.321 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:42:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:42:16.008+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:42:16.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:42:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:42:16.355+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:42:16.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:42:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:42:16.355+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:42:16.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:42:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:42:16.355+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:42:16.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:42:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:42:16.355+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:42:17.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:42:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:42:16.950+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:42:17.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:42:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:42:16.950+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:42:17.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:42:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:42:16.950+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:42:17.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:42:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:42:16.951+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:42:17.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:42:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:42:16.959+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:42:17.321 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:42:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:42:16.959+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:42:17.321 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:42:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:42:16.959+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:42:17.321 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:42:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:42:16.959+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:42:17.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:42:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:42:17.355+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:42:17.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:42:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:42:17.355+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:42:17.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:42:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:42:17.355+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:42:17.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:42:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:42:17.356+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:42:18.316 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:42:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:42:17.967+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:42:18.316 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:42:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:42:17.967+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:42:18.316 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:42:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:42:17.967+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:42:18.316 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:42:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:42:17.967+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:42:18.316 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:42:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:42:17.913+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:42:18.317 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:42:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:42:17.913+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:42:18.317 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:42:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:42:17.913+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:42:18.317 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:42:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:42:17.913+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:42:18.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:42:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:42:18.314+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:42:18.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:42:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:42:18.314+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:42:18.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:42:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:42:18.314+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:42:18.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:42:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:42:18.315+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:42:19.316 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:42:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:42:18.953+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:42:19.316 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:42:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:42:18.953+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:42:19.316 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:42:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:42:18.953+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:42:19.316 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:42:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:42:18.954+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:42:19.317 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:42:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:42:18.934+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:42:19.317 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:42:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:42:18.934+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:42:19.317 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:42:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:42:18.934+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:42:19.317 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:42:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:42:18.934+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:42:19.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:42:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:42:19.314+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:42:19.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:42:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:42:19.314+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:42:19.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:42:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:42:19.314+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:42:19.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:42:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:42:19.315+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:42:20.176 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:42:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:42:19.960+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:42:20.177 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:42:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:42:19.960+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:42:20.177 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:42:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:42:19.960+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:42:20.177 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:42:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:42:19.961+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:42:20.177 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:42:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:42:19.912+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:42:20.177 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:42:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:42:19.912+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:42:20.177 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:42:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:42:19.912+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:42:20.177 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:42:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:42:19.914+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:42:20.431 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:42:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:42:20.421+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 19549 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:42:20.431 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:42:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:42:20.288+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:42:20.431 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:42:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:42:20.288+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:42:20.431 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:42:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:42:20.288+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:42:20.431 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:42:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:42:20.289+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:42:21.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:42:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:42:20.877+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:42:21.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:42:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:42:20.877+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:42:21.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:42:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:42:20.877+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:42:21.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:42:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:42:20.877+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:42:21.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:42:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:42:20.926+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:42:21.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:42:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:42:20.926+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:42:21.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:42:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:42:20.926+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:42:21.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:42:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:42:20.926+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:42:21.708 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:42:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:42:21.330+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:42:21.708 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:42:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:42:21.330+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:42:21.708 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:42:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:42:21.330+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:42:21.708 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:42:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:42:21.330+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:42:22.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:42:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:42:21.865+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:42:22.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:42:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:42:21.865+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:42:22.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:42:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:42:21.865+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:42:22.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:42:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:42:21.866+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:42:22.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:42:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:42:21.914+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:42:22.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:42:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:42:21.914+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:42:22.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:42:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:42:21.915+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:42:22.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:42:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:42:21.915+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:42:22.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:42:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:42:22.323+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:42:22.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:42:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:42:22.323+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:42:22.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:42:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:42:22.323+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:42:22.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:42:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:42:22.324+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:42:23.285 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:42:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:42:22.824+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:42:23.285 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:42:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:42:22.824+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:42:23.285 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:42:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:42:22.824+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:42:23.285 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:42:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:42:22.825+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:42:23.286 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:42:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:42:22.949+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:42:23.286 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:42:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:42:22.949+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:42:23.286 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:42:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:42:22.949+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:42:23.286 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:42:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:42:22.949+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:42:23.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:42:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:42:23.284+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:42:23.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:42:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:42:23.284+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:42:23.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:42:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:42:23.284+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:42:23.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:42:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:42:23.285+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:42:24.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:42:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:42:23.850+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:42:24.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:42:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:42:23.850+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:42:24.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:42:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:42:23.850+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:42:24.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:42:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:42:23.850+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:42:24.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:42:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:42:23.975+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:42:24.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:42:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:42:23.975+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:42:24.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:42:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:42:23.975+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:42:24.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:42:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:42:23.975+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:42:24.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:42:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:42:24.255+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:42:24.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:42:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:42:24.255+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:42:24.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:42:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:42:24.255+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:42:24.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:42:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:42:24.255+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:42:25.283 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:42:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:42:24.981+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:42:25.284 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:42:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:42:24.981+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:42:25.284 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:42:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:42:24.981+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:42:25.284 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:42:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:42:24.981+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:42:25.284 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:42:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:42:24.836+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:42:25.284 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:42:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:42:24.836+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:42:25.284 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:42:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:42:24.836+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:42:25.284 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:42:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:42:24.837+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:42:25.547 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:42:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:42:25.423+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 19633 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:42:25.548 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:42:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:42:25.282+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:42:25.548 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:42:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:42:25.282+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:42:25.548 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:42:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:42:25.282+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:42:25.548 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:42:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:42:25.283+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:42:26.284 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:42:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:42:26.024+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:42:26.284 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:42:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:42:26.024+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:42:26.284 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:42:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:42:26.024+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:42:26.284 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:42:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:42:26.025+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:42:26.285 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:42:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:42:25.884+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:42:26.285 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:42:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:42:25.884+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:42:26.285 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:42:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:42:25.884+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:42:26.285 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:42:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:42:25.884+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:42:26.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:42:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:42:26.283+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:42:26.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:42:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:42:26.283+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:42:26.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:42:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:42:26.283+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:42:26.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:42:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:42:26.284+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:42:27.252 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:42:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:42:27.029+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:42:27.253 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:42:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:42:27.029+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:42:27.253 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:42:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:42:27.029+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:42:27.253 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:42:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:42:27.030+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:42:27.253 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:42:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:42:26.860+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:42:27.253 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:42:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:42:26.860+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:42:27.253 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:42:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:42:26.860+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:42:27.253 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:42:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:42:26.861+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:42:27.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:42:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:42:27.251+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:42:27.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:42:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:42:27.251+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:42:27.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:42:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:42:27.251+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:42:27.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:42:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:42:27.252+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:42:28.264 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:42:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:42:28.073+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:42:28.264 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:42:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:42:28.073+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:42:28.264 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:42:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:42:28.073+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:42:28.264 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:42:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:42:28.074+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:42:28.264 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:42:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:42:27.844+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:42:28.265 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:42:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:42:27.844+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:42:28.265 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:42:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:42:27.844+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:42:28.265 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:42:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:42:27.844+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:42:28.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:42:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:42:28.262+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:42:28.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:42:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:42:28.262+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:42:28.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:42:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:42:28.262+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:42:28.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:42:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:42:28.264+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:42:29.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:42:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:42:28.802+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:42:29.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:42:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:42:28.802+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:42:29.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:42:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:42:28.802+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:42:29.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:42:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:42:28.803+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:42:29.569 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:42:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:42:29.101+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:42:29.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:42:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:42:29.101+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:42:29.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:42:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:42:29.101+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:42:29.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:42:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:42:29.101+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:42:29.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:42:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:42:29.299+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:42:29.571 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:42:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:42:29.299+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:42:29.571 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:42:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:42:29.299+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:42:29.571 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:42:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:42:29.300+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:42:30.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:42:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:42:29.781+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:42:30.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:42:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:42:29.781+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:42:30.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:42:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:42:29.781+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:42:30.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:42:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:42:29.782+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:42:30.428 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:42:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:42:30.276+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:42:30.428 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:42:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:42:30.276+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:42:30.428 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:42:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:42:30.276+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:42:30.428 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:42:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:42:30.277+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:42:30.429 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:42:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:42:30.072+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:42:30.429 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:42:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:42:30.072+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:42:30.429 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:42:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:42:30.072+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:42:30.429 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:42:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:42:30.072+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:42:30.740 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:42:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:42:30.425+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 19715 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:42:31.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:42:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:42:30.739+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:42:31.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:42:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:42:30.739+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:42:31.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:42:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:42:30.739+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:42:31.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:42:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:42:30.740+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:42:31.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:42:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:42:31.284+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:42:31.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:42:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:42:31.284+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:42:31.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:42:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:42:31.284+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:42:31.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:42:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:42:31.284+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:42:31.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:42:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:42:31.120+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:42:31.571 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:42:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:42:31.120+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:42:31.571 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:42:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:42:31.120+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:42:31.571 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:42:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:42:31.121+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:42:32.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:42:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:42:31.765+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:42:32.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:42:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:42:31.765+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:42:32.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:42:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:42:31.765+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:42:32.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:42:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:42:31.766+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:42:32.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:42:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:42:32.294+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:42:32.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:42:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:42:32.294+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:42:32.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:42:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:42:32.294+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:42:32.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:42:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:42:32.295+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:42:32.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:42:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:42:32.152+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:42:32.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:42:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:42:32.152+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:42:32.571 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:42:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:42:32.152+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:42:32.571 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:42:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:42:32.152+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:42:32.988 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:42:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:42:32.717+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:42:32.988 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:42:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:42:32.717+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:42:32.988 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:42:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:42:32.717+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:42:32.988 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:42:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:42:32.718+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:42:33.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:42:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:42:33.119+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:42:33.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:42:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:42:33.119+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:42:33.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:42:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:42:33.119+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:42:33.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:42:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:42:33.120+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:42:33.715 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:42:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:42:33.337+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:42:33.715 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:42:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:42:33.337+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:42:33.715 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:42:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:42:33.337+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:42:33.715 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:42:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:42:33.338+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:42:34.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:42:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:42:33.714+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:42:34.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:42:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:42:33.714+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:42:34.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:42:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:42:33.714+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:42:34.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:42:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:42:33.715+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:42:34.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:42:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:42:34.336+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:42:34.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:42:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:42:34.336+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:42:34.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:42:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:42:34.336+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:42:34.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:42:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:42:34.336+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:42:34.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:42:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:42:34.126+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:42:34.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:42:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:42:34.126+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:42:34.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:42:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:42:34.126+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:42:34.571 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:42:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:42:34.127+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:42:35.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:42:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:42:34.733+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:42:35.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:42:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:42:34.733+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:42:35.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:42:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:42:34.733+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:42:35.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:42:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:42:34.734+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:42:35.428 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:42:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:42:35.362+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:42:35.429 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:42:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:42:35.362+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:42:35.429 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:42:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:42:35.362+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:42:35.429 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:42:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:42:35.363+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:42:35.429 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:42:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:42:35.118+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:42:35.429 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:42:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:42:35.118+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:42:35.429 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:42:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:42:35.118+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:42:35.429 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:42:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:42:35.119+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:42:35.727 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:42:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:42:35.427+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 19799 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:42:36.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:42:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:42:35.725+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:42:36.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:42:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:42:35.725+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:42:36.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:42:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:42:35.725+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:42:36.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:42:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:42:35.727+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:42:36.359 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:42:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:42:36.072+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:42:36.360 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:42:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:42:36.072+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:42:36.360 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:42:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:42:36.072+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:42:36.360 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:42:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:42:36.073+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:42:36.733 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:42:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:42:36.358+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:42:36.733 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:42:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:42:36.358+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:42:36.733 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:42:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:42:36.358+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:42:36.734 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:42:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:42:36.359+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:42:37.029 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:42:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:42:36.732+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:42:37.029 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:42:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:42:36.732+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:42:37.029 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:42:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:42:36.732+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:42:37.029 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:42:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:42:36.733+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:42:37.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:42:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:42:37.027+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:42:37.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:42:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:42:37.027+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:42:37.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:42:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:42:37.027+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:42:37.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:42:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:42:37.027+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:42:37.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:42:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:42:37.320+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:42:37.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:42:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:42:37.320+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:42:37.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:42:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:42:37.320+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:42:37.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:42:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:42:37.320+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:42:38.047 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:42:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:42:37.725+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:42:38.048 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:42:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:42:37.725+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:42:38.048 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:42:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:42:37.725+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:42:38.048 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:42:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:42:37.725+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:42:38.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:42:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:42:38.045+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:42:38.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:42:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:42:38.045+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:42:38.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:42:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:42:38.045+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:42:38.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:42:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:42:38.046+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:42:38.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:42:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:42:38.294+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:42:38.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:42:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:42:38.294+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:42:38.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:42:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:42:38.294+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:42:38.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:42:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:42:38.295+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:42:39.033 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:42:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:42:38.730+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:42:39.033 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:42:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:42:38.730+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:42:39.033 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:42:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:42:38.730+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:42:39.034 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:42:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:42:38.731+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:42:39.306 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:42:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:42:39.031+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:42:39.306 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:42:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:42:39.031+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:42:39.306 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:42:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:42:39.031+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:42:39.307 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:42:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:42:39.031+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:42:39.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:42:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:42:39.304+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:42:39.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:42:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:42:39.304+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:42:39.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:42:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:42:39.304+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:42:39.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:42:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:42:39.304+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:42:40.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:42:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:42:39.750+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:42:40.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:42:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:42:39.750+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:42:40.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:42:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:42:39.750+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:42:40.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:42:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:42:39.751+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:42:40.340 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:42:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:42:40.078+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:42:40.340 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:42:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:42:40.078+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:42:40.341 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:42:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:42:40.078+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:42:40.341 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:42:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:42:40.078+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:42:40.717 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:42:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:42:40.428+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 19878 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:42:40.717 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:42:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:42:40.338+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:42:40.717 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:42:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:42:40.338+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:42:40.717 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:42:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:42:40.338+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:42:40.718 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:42:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:42:40.339+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:42:41.047 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:42:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:42:40.715+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:42:41.047 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:42:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:42:40.715+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:42:41.048 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:42:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:42:40.715+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:42:41.048 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:42:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:42:40.716+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:42:41.303 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:42:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:42:41.045+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:42:41.304 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:42:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:42:41.045+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:42:41.304 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:42:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:42:41.045+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:42:41.304 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:42:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:42:41.045+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:42:41.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:42:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:42:41.301+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:42:41.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:42:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:42:41.301+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:42:41.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:42:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:42:41.301+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:42:41.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:42:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:42:41.302+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:42:42.057 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:42:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:42:41.748+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:42:42.057 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:42:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:42:41.748+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:42:42.057 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:42:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:42:41.748+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:42:42.057 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:42:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:42:41.749+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:42:42.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:42:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:42:42.054+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:42:42.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:42:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:42:42.054+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:42:42.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:42:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:42:42.054+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:42:42.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:42:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:42:42.055+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:42:42.754 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:42:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:42:42.341+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:42:42.754 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:42:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:42:42.341+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:42:42.754 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:42:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:42:42.341+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:42:42.754 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:42:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:42:42.342+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:42:43.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:42:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:42:42.751+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:42:43.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:42:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:42:42.751+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:42:43.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:42:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:42:42.751+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:42:43.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:42:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:42:42.752+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:42:43.352 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:42:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:42:43.082+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:42:43.352 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:42:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:42:43.082+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:42:43.353 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:42:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:42:43.082+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:42:43.353 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:42:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:42:43.082+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:42:43.725 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:42:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:42:43.350+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:42:43.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:42:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:42:43.350+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:42:43.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:42:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:42:43.350+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:42:43.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:42:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:42:43.351+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:42:44.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:42:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:42:43.723+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:42:44.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:42:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:42:43.723+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:42:44.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:42:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:42:43.723+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:42:44.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:42:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:42:43.724+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:42:44.331 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:42:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:42:44.131+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:42:44.331 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:42:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:42:44.131+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:42:44.331 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:42:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:42:44.131+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:42:44.331 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:42:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:42:44.132+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:42:44.738 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:42:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:42:44.329+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:42:44.738 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:42:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:42:44.329+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:42:44.739 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:42:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:42:44.329+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:42:44.739 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:42:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:42:44.330+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:42:45.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:42:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:42:44.736+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:42:45.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:42:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:42:44.736+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:42:45.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:42:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:42:44.736+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:42:45.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:42:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:42:44.737+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:42:45.433 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:42:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:42:45.334+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:42:45.433 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:42:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:42:45.334+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:42:45.433 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:42:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:42:45.334+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:42:45.433 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:42:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:42:45.335+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:42:45.434 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:42:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:42:45.121+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:42:45.434 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:42:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:42:45.121+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:42:45.434 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:42:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:42:45.121+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:42:45.434 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:42:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:42:45.121+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:42:45.763 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:42:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:42:45.430+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 19959 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:42:46.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:42:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:42:45.761+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:42:46.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:42:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:42:45.761+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:42:46.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:42:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:42:45.761+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:42:46.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:42:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:42:45.762+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:42:46.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:42:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:42:46.383+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:42:46.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:42:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:42:46.383+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:42:46.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:42:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:42:46.383+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:42:46.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:42:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:42:46.383+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:42:46.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:42:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:42:46.155+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:42:46.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:42:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:42:46.155+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:42:46.571 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:42:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:42:46.155+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:42:46.571 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:42:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:42:46.155+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:42:47.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:42:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:42:46.808+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:42:47.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:42:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:42:46.809+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:42:47.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:42:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:42:46.809+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:42:47.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:42:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:42:46.810+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:42:47.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:42:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:42:47.401+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:42:47.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:42:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:42:47.401+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:42:47.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:42:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:42:47.401+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:42:47.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:42:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:42:47.401+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:42:47.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:42:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:42:47.177+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:42:47.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:42:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:42:47.177+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:42:47.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:42:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:42:47.177+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:42:47.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:42:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:42:47.177+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:42:48.225 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:42:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:42:47.831+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:42:48.225 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:42:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:42:47.831+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:42:48.225 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:42:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:42:47.831+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:42:48.225 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:42:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:42:47.833+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:42:48.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:42:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:42:48.402+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:42:48.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:42:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:42:48.402+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:42:48.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:42:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:42:48.402+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:42:48.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:42:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:42:48.403+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:42:48.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:42:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:42:48.224+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:42:48.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:42:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:42:48.224+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:42:48.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:42:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:42:48.224+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:42:48.571 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:42:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:42:48.224+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:42:49.116 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:42:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:42:48.819+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:42:49.116 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:42:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:42:48.819+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:42:49.116 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:42:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:42:48.819+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:42:49.116 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:42:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:42:48.820+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:42:49.441 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:42:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:42:49.262+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:42:49.441 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:42:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:42:49.262+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:42:49.441 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:42:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:42:49.262+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:42:49.441 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:42:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:42:49.262+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:42:49.795 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:42:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:42:49.439+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:42:49.795 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:42:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:42:49.440+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:42:49.795 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:42:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:42:49.440+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:42:49.795 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:42:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:42:49.440+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:42:50.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:42:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:42:49.794+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:42:50.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:42:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:42:49.794+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:42:50.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:42:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:42:49.794+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:42:50.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:42:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:42:49.795+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:42:50.437 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:42:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:42:50.265+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:42:50.437 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:42:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:42:50.265+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:42:50.437 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:42:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:42:50.265+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:42:50.438 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:42:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:42:50.265+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:42:50.438 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:42:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:42:50.415+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:42:50.438 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:42:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:42:50.415+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:42:50.438 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:42:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:42:50.415+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:42:50.438 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:42:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:42:50.415+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:42:50.785 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:42:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:42:50.434+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 20038 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:42:51.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:42:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:42:50.784+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:42:51.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:42:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:42:50.784+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:42:51.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:42:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:42:50.784+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:42:51.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:42:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:42:50.785+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:42:51.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:42:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:42:51.389+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:42:51.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:42:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:42:51.389+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:42:51.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:42:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:42:51.389+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:42:51.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:42:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:42:51.389+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:42:51.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:42:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:42:51.228+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:42:51.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:42:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:42:51.228+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:42:51.571 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:42:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:42:51.228+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:42:51.571 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:42:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:42:51.228+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:42:52.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:42:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:42:51.805+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:42:52.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:42:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:42:51.805+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:42:52.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:42:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:42:51.805+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:42:52.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:42:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:42:51.807+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:42:52.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:42:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:42:52.418+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:42:52.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:42:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:42:52.418+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:42:52.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:42:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:42:52.418+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:42:52.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:42:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:42:52.419+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:42:52.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:42:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:42:52.260+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:42:52.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:42:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:42:52.260+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:42:52.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:42:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:42:52.260+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:42:52.571 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:42:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:42:52.260+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:42:53.241 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:42:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:42:52.824+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:42:53.241 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:42:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:42:52.824+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:42:53.241 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:42:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:42:52.824+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:42:53.241 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:42:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:42:52.825+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:42:53.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:42:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:42:53.444+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:42:53.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:42:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:42:53.444+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:42:53.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:42:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:42:53.444+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:42:53.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:42:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:42:53.445+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:42:53.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:42:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:42:53.239+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:42:53.571 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:42:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:42:53.239+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:42:53.571 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:42:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:42:53.239+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:42:53.571 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:42:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:42:53.240+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:42:54.242 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:42:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:42:53.861+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:42:54.242 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:42:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:42:53.861+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:42:54.242 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:42:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:42:53.861+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:42:54.242 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:42:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:42:53.861+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:42:54.494 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:42:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:42:54.241+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:42:54.495 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:42:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:42:54.241+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:42:54.495 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:42:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:42:54.241+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:42:54.495 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:42:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:42:54.241+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:42:54.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:42:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:42:54.493+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:42:54.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:42:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:42:54.493+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:42:54.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:42:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:42:54.493+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:42:54.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:42:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:42:54.494+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:42:55.225 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:42:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:42:54.829+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:42:55.225 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:42:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:42:54.829+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:42:55.226 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:42:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:42:54.829+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:42:55.226 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:42:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:42:54.829+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:42:55.501 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:42:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:42:55.436+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 20113 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:42:55.501 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:42:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:42:55.224+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:42:55.501 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:42:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:42:55.224+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:42:55.501 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:42:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:42:55.224+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:42:55.502 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:42:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:42:55.225+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:42:55.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:42:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:42:55.500+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:42:55.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:42:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:42:55.500+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:42:55.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:42:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:42:55.500+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:42:55.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:42:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:42:55.501+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:42:56.197 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:42:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:42:55.843+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:42:56.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:42:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:42:55.843+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:42:56.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:42:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:42:55.843+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:42:56.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:42:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:42:55.843+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:42:56.495 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:42:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:42:56.196+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:42:56.495 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:42:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:42:56.196+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:42:56.495 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:42:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:42:56.196+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:42:56.495 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:42:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:42:56.197+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:42:56.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:42:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:42:56.493+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:42:56.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:42:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:42:56.493+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:42:56.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:42:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:42:56.493+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:42:56.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:42:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:42:56.494+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:42:57.178 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:42:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:42:56.845+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:42:57.178 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:42:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:42:56.845+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:42:57.178 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:42:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:42:56.845+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:42:57.178 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:42:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:42:56.846+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:42:57.535 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:42:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:42:57.177+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:42:57.535 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:42:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:42:57.177+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:42:57.535 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:42:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:42:57.177+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:42:57.535 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:42:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:42:57.177+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:42:57.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:42:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:42:57.534+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:42:57.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:42:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:42:57.534+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:42:57.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:42:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:42:57.534+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:42:57.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:42:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:42:57.534+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:42:58.211 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:42:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:42:57.819+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:42:58.211 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:42:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:42:57.819+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:42:58.211 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:42:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:42:57.819+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:42:58.211 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:42:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:42:57.821+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:42:58.569 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:42:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:42:58.209+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:42:58.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:42:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:42:58.209+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:42:58.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:42:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:42:58.209+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:42:58.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:42:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:42:58.210+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:42:58.867 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:42:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:42:58.581+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:42:58.867 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:42:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:42:58.581+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:42:58.867 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:42:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:42:58.581+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:42:58.867 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:42:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:42:58.582+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:42:59.202 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:42:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:42:58.866+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:42:59.202 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:42:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:42:58.866+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:42:59.202 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:42:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:42:58.866+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:42:59.202 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:42:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:42:58.867+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:42:59.569 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:42:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:42:59.200+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:42:59.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:42:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:42:59.200+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:42:59.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:42:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:42:59.200+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:42:59.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:42:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:42:59.201+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:43:00.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:42:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:42:59.835+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:43:00.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:42:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:42:59.835+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:43:00.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:42:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:42:59.835+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:43:00.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:42:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:42:59.837+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:43:00.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:42:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:42:59.600+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:43:00.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:42:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:42:59.600+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:43:00.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:42:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:42:59.600+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:43:00.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:42:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:42:59.600+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:43:00.439 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:43:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:43:00.167+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:43:00.439 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:43:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:43:00.167+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:43:00.439 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:43:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:43:00.167+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:43:00.439 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:43:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:43:00.168+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:43:00.819 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:43:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:43:00.438+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 20189 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:43:00.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:43:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:43:00.630+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:43:00.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:43:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:43:00.630+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:43:00.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:43:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:43:00.630+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:43:00.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:43:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:43:00.631+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:43:01.181 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:43:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:43:00.879+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:43:01.181 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:43:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:43:00.879+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:43:01.181 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:43:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:43:00.879+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:43:01.181 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:43:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:43:00.880+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:43:01.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:43:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:43:01.180+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:43:01.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:43:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:43:01.180+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:43:01.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:43:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:43:01.180+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:43:01.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:43:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:43:01.180+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:43:02.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:43:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:43:01.667+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:43:02.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:43:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:43:01.667+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:43:02.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:43:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:43:01.667+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:43:02.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:43:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:43:01.668+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:43:02.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:43:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:43:01.898+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:43:02.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:43:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:43:01.898+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:43:02.071 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:43:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:43:01.898+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:43:02.071 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:43:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:43:01.899+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:43:02.569 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:43:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:43:02.152+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:43:02.569 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:43:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:43:02.152+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:43:02.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:43:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:43:02.152+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:43:02.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:43:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:43:02.152+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:43:03.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:43:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:43:02.673+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:43:03.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:43:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:43:02.673+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:43:03.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:43:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:43:02.673+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:43:03.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:43:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:43:02.674+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:43:03.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:43:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:43:02.900+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:43:03.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:43:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:43:02.900+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:43:03.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:43:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:43:02.900+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:43:03.071 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:43:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:43:02.901+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:43:03.569 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:43:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:43:03.118+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:43:03.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:43:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:43:03.118+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:43:03.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:43:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:43:03.118+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:43:03.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:43:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:43:03.119+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:43:04.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:43:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:43:03.943+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:43:04.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:43:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:43:03.943+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:43:04.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:43:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:43:03.943+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:43:04.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:43:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:43:03.944+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:43:04.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:43:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:43:03.705+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:43:04.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:43:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:43:03.705+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:43:04.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:43:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:43:03.705+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:43:04.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:43:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:43:03.706+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:43:04.569 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:43:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:43:04.095+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:43:04.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:43:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:43:04.095+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:43:04.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:43:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:43:04.095+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:43:04.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:43:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:43:04.096+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:43:04.908 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:43:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:43:04.656+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:43:04.909 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:43:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:43:04.656+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:43:04.909 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:43:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:43:04.656+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:43:04.909 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:43:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:43:04.657+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:43:05.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:43:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:43:04.907+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:43:05.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:43:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:43:04.907+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:43:05.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:43:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:43:04.907+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:43:05.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:43:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:43:04.908+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:43:05.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:43:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:43:05.131+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:43:05.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:43:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:43:05.132+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:43:05.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:43:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:43:05.132+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:43:05.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:43:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:43:05.132+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:43:05.819 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:43:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:43:05.440+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 20267 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:43:05.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:43:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:43:05.654+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:43:05.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:43:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:43:05.654+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:43:05.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:43:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:43:05.654+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:43:05.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:43:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:43:05.655+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:43:06.129 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:43:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:43:05.859+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:43:06.130 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:43:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:43:05.859+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:43:06.130 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:43:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:43:05.859+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:43:06.130 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:43:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:43:05.860+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:43:06.569 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:43:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:43:06.128+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:43:06.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:43:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:43:06.128+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:43:06.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:43:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:43:06.128+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:43:06.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:43:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:43:06.128+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:43:07.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:43:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:43:06.838+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:43:07.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:43:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:43:06.838+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:43:07.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:43:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:43:06.838+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:43:07.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:43:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:43:06.839+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:43:07.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:43:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:43:06.654+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:43:07.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:43:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:43:06.654+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:43:07.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:43:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:43:06.654+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:43:07.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:43:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:43:06.654+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:43:07.569 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:43:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:43:07.112+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:43:07.569 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:43:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:43:07.112+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:43:07.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:43:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:43:07.112+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:43:07.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:43:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:43:07.113+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:43:08.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:43:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:43:07.796+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:43:08.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:43:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:43:07.796+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:43:08.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:43:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:43:07.796+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:43:08.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:43:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:43:07.796+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:43:08.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:43:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:43:07.690+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:43:08.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:43:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:43:07.690+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:43:08.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:43:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:43:07.690+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:43:08.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:43:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:43:07.690+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:43:08.569 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:43:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:43:08.157+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:43:08.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:43:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:43:08.157+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:43:08.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:43:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:43:08.157+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:43:08.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:43:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:43:08.157+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:43:09.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:43:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:43:08.780+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:43:09.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:43:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:43:08.780+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:43:09.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:43:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:43:08.780+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:43:09.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:43:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:43:08.781+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:43:09.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:43:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:43:08.670+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:43:09.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:43:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:43:08.670+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:43:09.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:43:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:43:08.670+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:43:09.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:43:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:43:08.670+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:43:09.569 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:43:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:43:09.172+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:43:09.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:43:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:43:09.172+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:43:09.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:43:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:43:09.172+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:43:09.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:43:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:43:09.173+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:43:10.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:43:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:43:09.803+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:43:10.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:43:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:43:09.803+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:43:10.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:43:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:43:09.803+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:43:10.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:43:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:43:09.804+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:43:10.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:43:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:43:09.686+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:43:10.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:43:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:43:09.686+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:43:10.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:43:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:43:09.686+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:43:10.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:43:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:43:09.687+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:43:10.443 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:43:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:43:10.204+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:43:10.443 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:43:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:43:10.204+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:43:10.443 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:43:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:43:10.204+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:43:10.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:43:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:43:10.204+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:43:10.710 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:43:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:43:10.442+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 20343 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:43:11.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:43:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:43:10.764+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:43:11.071 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:43:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:43:10.764+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:43:11.071 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:43:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:43:10.764+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:43:11.071 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:43:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:43:10.765+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:43:11.072 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:43:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:43:10.709+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:43:11.072 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:43:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:43:10.709+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:43:11.072 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:43:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:43:10.709+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:43:11.073 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:43:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:43:10.710+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:43:11.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:43:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:43:11.212+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:43:11.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:43:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:43:11.212+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:43:11.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:43:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:43:11.212+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:43:11.571 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:43:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:43:11.213+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:43:12.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:43:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:43:11.729+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:43:12.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:43:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:43:11.729+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:43:12.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:43:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:43:11.729+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:43:12.071 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:43:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:43:11.730+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:43:12.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:43:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:43:11.741+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:43:12.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:43:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:43:11.741+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:43:12.072 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:43:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:43:11.741+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:43:12.072 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:43:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:43:11.742+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:43:12.569 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:43:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:43:12.207+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:43:12.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:43:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:43:12.207+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:43:12.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:43:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:43:12.207+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:43:12.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:43:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:43:12.207+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:43:13.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:43:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:43:12.769+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:43:13.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:43:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:43:12.769+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:43:13.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:43:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:43:12.769+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:43:13.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:43:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:43:12.770+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:43:13.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:43:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:43:12.712+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:43:13.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:43:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:43:12.712+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:43:13.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:43:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:43:12.712+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:43:13.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:43:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:43:12.713+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:43:13.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:43:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:43:13.218+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:43:13.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:43:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:43:13.218+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:43:13.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:43:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:43:13.218+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:43:13.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:43:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:43:13.219+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:43:14.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:43:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:43:13.741+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:43:14.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:43:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:43:13.741+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:43:14.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:43:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:43:13.741+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:43:14.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:43:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:43:13.741+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:43:14.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:43:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:43:13.708+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:43:14.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:43:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:43:13.708+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:43:14.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:43:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:43:13.708+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:43:14.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:43:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:43:13.709+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:43:14.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:43:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:43:14.264+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:43:14.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:43:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:43:14.264+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:43:14.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:43:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:43:14.264+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:43:14.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:43:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:43:14.264+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:43:15.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:43:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:43:14.730+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:43:15.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:43:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:43:14.730+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:43:15.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:43:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:43:14.730+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:43:15.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:43:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:43:14.730+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:43:15.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:43:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:43:14.665+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:43:15.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:43:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:43:14.665+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:43:15.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:43:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:43:14.665+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:43:15.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:43:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:43:14.666+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:43:15.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:43:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:43:15.444+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 20427 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:43:15.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:43:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:43:15.241+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:43:15.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:43:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:43:15.241+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:43:15.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:43:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:43:15.241+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:43:15.571 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:43:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:43:15.241+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:43:16.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:43:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:43:15.728+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:43:16.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:43:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:43:15.728+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:43:16.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:43:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:43:15.728+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:43:16.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:43:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:43:15.728+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:43:16.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:43:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:43:15.675+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:43:16.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:43:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:43:15.675+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:43:16.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:43:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:43:15.675+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:43:16.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:43:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:43:15.676+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:43:16.569 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:43:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:43:16.227+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:43:16.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:43:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:43:16.227+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:43:16.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:43:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:43:16.227+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:43:16.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:43:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:43:16.227+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:43:17.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:43:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:43:16.765+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:43:17.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:43:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:43:16.765+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:43:17.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:43:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:43:16.765+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:43:17.071 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:43:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:43:16.766+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:43:17.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:43:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:43:16.712+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:43:17.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:43:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:43:16.712+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:43:17.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:43:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:43:16.712+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:43:17.072 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:43:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:43:16.712+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:43:17.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:43:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:43:17.183+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:43:17.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:43:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:43:17.183+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:43:17.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:43:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:43:17.183+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:43:17.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:43:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:43:17.184+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:43:18.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:43:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:43:17.768+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:43:18.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:43:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:43:17.768+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:43:18.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:43:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:43:17.768+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:43:18.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:43:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:43:17.769+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:43:18.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:43:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:43:17.711+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:43:18.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:43:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:43:17.711+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:43:18.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:43:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:43:17.711+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:43:18.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:43:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:43:17.712+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:43:18.569 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:43:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:43:18.205+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:43:18.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:43:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:43:18.205+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:43:18.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:43:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:43:18.205+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:43:18.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:43:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:43:18.205+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:43:19.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:43:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:43:18.686+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:43:19.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:43:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:43:18.686+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:43:19.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:43:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:43:18.686+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:43:19.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:43:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:43:18.686+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:43:19.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:43:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:43:18.720+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:43:19.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:43:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:43:18.720+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:43:19.071 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:43:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:43:18.720+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:43:19.071 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:43:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:43:18.722+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:43:19.569 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:43:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:43:19.239+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:43:19.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:43:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:43:19.239+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:43:19.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:43:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:43:19.239+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:43:19.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:43:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:43:19.239+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:43:20.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:43:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:43:19.675+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:43:20.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:43:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:43:19.675+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:43:20.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:43:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:43:19.675+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:43:20.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:43:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:43:19.676+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:43:20.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:43:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:43:19.718+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:43:20.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:43:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:43:19.718+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:43:20.071 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:43:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:43:19.718+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:43:20.071 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:43:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:43:19.718+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:43:20.448 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:43:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:43:20.202+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:43:20.448 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:43:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:43:20.202+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:43:20.448 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:43:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:43:20.202+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:43:20.448 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:43:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:43:20.203+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:43:20.734 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:43:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:43:20.446+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 20509 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:43:20.735 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:43:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:43:20.669+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:43:20.735 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:43:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:43:20.669+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:43:20.735 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:43:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:43:20.669+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:43:20.735 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:43:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:43:20.669+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:43:21.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:43:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:43:20.733+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:43:21.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:43:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:43:20.733+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:43:21.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:43:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:43:20.733+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:43:21.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:43:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:43:20.734+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:43:21.569 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:43:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:43:21.252+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:43:21.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:43:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:43:21.252+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:43:21.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:43:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:43:21.252+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:43:21.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:43:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:43:21.253+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:43:22.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:43:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:43:21.689+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:43:22.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:43:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:43:21.689+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:43:22.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:43:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:43:21.689+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:43:22.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:43:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:43:21.690+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:43:22.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:43:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:43:21.690+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:43:22.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:43:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:43:21.690+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:43:22.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:43:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:43:21.690+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:43:22.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:43:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:43:21.691+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:43:22.569 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:43:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:43:22.259+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:43:22.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:43:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:43:22.259+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:43:22.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:43:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:43:22.259+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:43:22.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:43:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:43:22.259+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:43:23.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:43:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:43:22.716+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:43:23.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:43:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:43:22.716+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:43:23.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:43:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:43:22.716+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:43:23.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:43:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:43:22.717+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:43:23.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:43:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:43:22.675+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:43:23.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:43:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:43:22.675+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:43:23.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:43:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:43:22.675+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:43:23.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:43:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:43:22.675+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:43:23.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:43:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:43:23.218+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:43:23.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:43:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:43:23.218+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:43:23.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:43:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:43:23.218+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:43:23.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:43:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:43:23.219+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:43:24.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:43:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:43:23.738+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:43:24.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:43:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:43:23.738+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:43:24.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:43:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:43:23.738+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:43:24.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:43:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:43:23.738+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:43:24.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:43:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:43:23.711+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:43:24.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:43:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:43:23.711+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:43:24.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:43:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:43:23.711+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:43:24.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:43:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:43:23.712+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:43:24.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:43:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:43:24.173+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:43:24.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:43:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:43:24.173+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:43:24.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:43:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:43:24.173+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:43:24.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:43:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:43:24.173+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:43:25.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:43:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:43:24.745+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:43:25.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:43:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:43:24.745+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:43:25.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:43:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:43:24.745+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:43:25.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:43:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:43:24.747+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:43:25.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:43:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:43:24.712+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:43:25.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:43:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:43:24.712+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:43:25.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:43:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:43:24.712+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:43:25.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:43:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:43:24.713+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:43:25.450 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:43:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:43:25.128+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:43:25.450 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:43:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:43:25.128+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:43:25.451 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:43:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:43:25.128+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:43:25.451 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:43:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:43:25.128+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:43:25.703 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:43:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:43:25.449+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 20593 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:43:26.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:43:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:43:25.738+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:43:26.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:43:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:43:25.738+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:43:26.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:43:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:43:25.738+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:43:26.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:43:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:43:25.739+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:43:26.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:43:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:43:25.702+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:43:26.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:43:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:43:25.702+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:43:26.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:43:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:43:25.702+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:43:26.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:43:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:43:25.703+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:43:26.569 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:43:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:43:26.122+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:43:26.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:43:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:43:26.122+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:43:26.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:43:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:43:26.122+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:43:26.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:43:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:43:26.122+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:43:27.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:43:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:43:26.707+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:43:27.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:43:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:43:26.707+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:43:27.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:43:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:43:26.707+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:43:27.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:43:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:43:26.708+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:43:27.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:43:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:43:26.664+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:43:27.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:43:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:43:26.664+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:43:27.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:43:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:43:26.664+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:43:27.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:43:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:43:26.665+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:43:27.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:43:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:43:27.143+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:43:27.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:43:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:43:27.143+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:43:27.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:43:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:43:27.143+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:43:27.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:43:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:43:27.144+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:43:28.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:43:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:43:27.709+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:43:28.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:43:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:43:27.709+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:43:28.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:43:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:43:27.709+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:43:28.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:43:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:43:27.710+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:43:28.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:43:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:43:27.624+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:43:28.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:43:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:43:27.624+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:43:28.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:43:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:43:27.624+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:43:28.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:43:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:43:27.625+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:43:28.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:43:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:43:28.127+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:43:28.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:43:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:43:28.127+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:43:28.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:43:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:43:28.127+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:43:28.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:43:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:43:28.128+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:43:29.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:43:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:43:28.703+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:43:29.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:43:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:43:28.703+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:43:29.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:43:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:43:28.703+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:43:29.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:43:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:43:28.704+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:43:29.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:43:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:43:28.616+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:43:29.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:43:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:43:28.616+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:43:29.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:43:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:43:28.616+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:43:29.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:43:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:43:28.617+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:43:29.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:43:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:43:29.125+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:43:29.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:43:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:43:29.125+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:43:29.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:43:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:43:29.125+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:43:29.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:43:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:43:29.126+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:43:30.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:43:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:43:29.679+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:43:30.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:43:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:43:29.679+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:43:30.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:43:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:43:29.679+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:43:30.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:43:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:43:29.680+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:43:30.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:43:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:43:29.630+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:43:30.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:43:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:43:29.630+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:43:30.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:43:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:43:29.631+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:43:30.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:43:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:43:29.631+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:43:30.452 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:43:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:43:30.175+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:43:30.452 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:43:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:43:30.175+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:43:30.452 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:43:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:43:30.175+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:43:30.453 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:43:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:43:30.175+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:43:30.819 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:43:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:43:30.451+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 20675 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:43:30.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:43:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:43:30.665+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:43:30.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:43:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:43:30.665+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:43:30.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:43:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:43:30.665+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:43:30.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:43:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:43:30.666+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:43:30.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:43:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:43:30.661+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:43:30.821 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:43:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:43:30.661+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:43:30.821 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:43:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:43:30.661+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:43:30.821 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:43:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:43:30.662+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:43:31.569 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:43:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:43:31.199+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:43:31.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:43:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:43:31.199+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:43:31.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:43:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:43:31.199+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:43:31.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:43:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:43:31.199+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:43:32.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:43:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:43:31.633+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:43:32.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:43:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:43:31.633+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:43:32.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:43:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:43:31.633+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:43:32.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:43:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:43:31.634+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:43:32.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:43:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:43:31.639+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:43:32.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:43:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:43:31.639+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:43:32.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:43:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:43:31.639+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:43:32.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:43:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:43:31.639+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:43:32.569 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:43:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:43:32.159+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:43:32.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:43:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:43:32.159+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:43:32.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:43:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:43:32.159+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:43:32.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:43:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:43:32.160+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:43:33.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:43:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:43:32.671+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:43:33.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:43:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:43:32.671+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:43:33.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:43:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:43:32.671+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:43:33.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:43:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:43:32.671+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:43:33.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:43:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:43:32.618+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:43:33.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:43:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:43:32.618+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:43:33.071 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:43:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:43:32.618+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:43:33.071 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:43:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:43:32.619+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:43:33.569 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:43:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:43:33.132+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:43:33.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:43:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:43:33.132+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:43:33.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:43:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:43:33.132+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:43:33.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:43:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:43:33.133+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:43:34.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:43:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:43:33.661+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:43:34.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:43:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:43:33.661+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:43:34.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:43:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:43:33.661+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:43:34.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:43:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:43:33.662+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:43:34.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:43:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:43:33.629+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:43:34.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:43:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:43:33.629+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:43:34.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:43:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:43:33.629+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:43:34.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:43:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:43:33.630+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:43:34.427 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:43:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:43:34.113+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:43:34.427 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:43:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:43:34.113+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:43:34.427 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:43:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:43:34.113+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:43:34.427 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:43:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:43:34.113+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:43:34.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:43:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:43:34.584+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:43:34.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:43:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:43:34.584+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:43:34.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:43:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:43:34.584+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:43:34.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:43:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:43:34.585+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:43:34.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:43:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:43:34.632+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:43:34.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:43:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:43:34.632+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:43:34.821 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:43:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:43:34.632+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:43:34.821 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:43:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:43:34.633+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:43:35.454 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:43:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:43:35.105+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:43:35.455 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:43:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:43:35.105+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:43:35.455 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:43:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:43:35.105+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:43:35.455 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:43:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:43:35.106+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:43:35.819 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:43:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:43:35.453+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 20759 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:43:35.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:43:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:43:35.552+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:43:35.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:43:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:43:35.552+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:43:35.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:43:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:43:35.552+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:43:35.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:43:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:43:35.553+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:43:35.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:43:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:43:35.593+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:43:35.821 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:43:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:43:35.593+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:43:35.821 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:43:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:43:35.593+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:43:35.821 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:43:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:43:35.594+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:43:36.542 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:43:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:43:36.111+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:43:36.543 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:43:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:43:36.111+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:43:36.543 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:43:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:43:36.111+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:43:36.543 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:43:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:43:36.111+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:43:36.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:43:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:43:36.540+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:43:36.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:43:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:43:36.540+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:43:36.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:43:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:43:36.540+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:43:36.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:43:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:43:36.541+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:43:36.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:43:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:43:36.577+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:43:36.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:43:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:43:36.577+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:43:36.821 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:43:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:43:36.577+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:43:36.821 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:43:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:43:36.577+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:43:37.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:43:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:43:37.065+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:43:37.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:43:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:43:37.065+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:43:37.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:43:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:43:37.065+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:43:37.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:43:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:43:37.066+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:43:37.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:43:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:43:37.560+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:43:37.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:43:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:43:37.560+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:43:37.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:43:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:43:37.560+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:43:37.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:43:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:43:37.560+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:43:37.821 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:43:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:43:37.600+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:43:37.821 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:43:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:43:37.600+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:43:37.821 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:43:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:43:37.600+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:43:37.821 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:43:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:43:37.600+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:43:38.569 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:43:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:43:38.081+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:43:38.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:43:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:43:38.081+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:43:38.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:43:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:43:38.081+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:43:38.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:43:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:43:38.081+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:43:39.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:43:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:43:38.581+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:43:39.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:43:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:43:38.581+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:43:39.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:43:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:43:38.581+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:43:39.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:43:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:43:38.581+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:43:39.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:43:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:43:38.623+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:43:39.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:43:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:43:38.623+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:43:39.071 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:43:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:43:38.623+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:43:39.071 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:43:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:43:38.624+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:43:39.569 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:43:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:43:39.093+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:43:39.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:43:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:43:39.093+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:43:39.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:43:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:43:39.093+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:43:39.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:43:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:43:39.093+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:43:40.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:43:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:43:39.617+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:43:40.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:43:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:43:39.617+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:43:40.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:43:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:43:39.617+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:43:40.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:43:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:43:39.618+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:43:40.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:43:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:43:39.649+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:43:40.071 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:43:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:43:39.649+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:43:40.071 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:43:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:43:39.649+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:43:40.071 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:43:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:43:39.650+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:43:40.457 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:43:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:43:40.073+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:43:40.457 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:43:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:43:40.073+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:43:40.457 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:43:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:43:40.073+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:43:40.457 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:43:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:43:40.073+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:43:40.819 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:43:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:43:40.454+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 20841 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:43:40.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:43:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:43:40.627+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:43:40.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:43:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:43:40.627+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:43:40.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:43:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:43:40.627+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:43:40.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:43:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:43:40.628+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:43:40.821 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:43:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:43:40.690+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:43:40.821 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:43:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:43:40.690+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:43:40.821 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:43:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:43:40.690+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:43:40.821 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:43:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:43:40.690+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:43:41.569 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:43:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:43:41.084+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:43:41.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:43:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:43:41.084+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:43:41.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:43:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:43:41.084+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:43:41.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:43:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:43:41.084+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:43:42.038 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:43:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:43:41.703+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:43:42.038 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:43:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:43:41.703+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:43:42.038 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:43:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:43:41.703+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:43:42.039 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:43:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:43:41.703+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:43:42.039 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:43:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:43:41.637+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:43:42.039 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:43:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:43:41.637+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:43:42.039 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:43:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:43:41.637+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:43:42.039 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:43:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:43:41.638+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:43:42.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:43:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:43:42.036+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:43:42.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:43:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:43:42.036+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:43:42.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:43:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:43:42.036+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:43:42.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:43:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:43:42.036+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:43:42.997 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:43:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:43:42.644+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:43:42.997 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:43:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:43:42.644+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:43:42.997 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:43:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:43:42.644+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:43:42.997 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:43:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:43:42.645+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:43:42.998 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:43:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:43:42.674+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:43:42.998 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:43:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:43:42.674+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:43:42.998 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:43:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:43:42.674+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:43:42.998 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:43:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:43:42.674+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:43:43.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:43:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:43:42.995+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:43:43.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:43:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:43:42.995+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:43:43.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:43:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:43:42.995+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:43:43.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:43:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:43:42.995+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:43:44.044 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:43:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:43:43.664+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:43:44.044 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:43:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:43:43.664+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:43:44.045 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:43:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:43:43.664+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:43:44.045 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:43:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:43:43.665+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:43:44.045 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:43:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:43:43.620+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:43:44.045 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:43:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:43:43.620+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:43:44.045 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:43:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:43:43.620+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:43:44.045 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:43:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:43:43.620+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:43:44.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:43:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:43:44.042+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:43:44.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:43:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:43:44.042+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:43:44.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:43:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:43:44.042+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:43:44.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:43:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:43:44.043+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:43:45.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:43:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:43:44.681+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:43:45.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:43:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:43:44.681+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:43:45.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:43:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:43:44.681+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:43:45.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:43:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:43:44.682+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:43:45.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:43:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:43:44.647+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:43:45.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:43:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:43:44.647+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:43:45.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:43:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:43:44.647+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:43:45.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:43:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:43:44.647+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:43:45.458 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:43:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:43:45.081+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:43:45.459 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:43:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:43:45.081+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:43:45.459 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:43:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:43:45.081+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:43:45.459 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:43:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:43:45.081+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:43:45.819 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:43:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:43:45.456+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 20925 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:43:45.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:43:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:43:45.683+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:43:45.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:43:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:43:45.683+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:43:45.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:43:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:43:45.683+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:43:45.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:43:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:43:45.683+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:43:45.821 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:43:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:43:45.692+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:43:45.821 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:43:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:43:45.692+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:43:45.821 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:43:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:43:45.692+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:43:45.821 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:43:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:43:45.693+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:43:46.569 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:43:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:43:46.118+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:43:46.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:43:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:43:46.118+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:43:46.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:43:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:43:46.118+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:43:46.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:43:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:43:46.118+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:43:47.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:43:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:43:46.665+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:43:47.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:43:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:43:46.665+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:43:47.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:43:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:43:46.665+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:43:47.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:43:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:43:46.666+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:43:47.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:43:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:43:46.720+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:43:47.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:43:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:43:46.720+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:43:47.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:43:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:43:46.720+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:43:47.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:43:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:43:46.721+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:43:47.569 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:43:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:43:47.153+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:43:47.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:43:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:43:47.153+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:43:47.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:43:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:43:47.153+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:43:47.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:43:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:43:47.153+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:43:48.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:43:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:43:47.692+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:43:48.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:43:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:43:47.692+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:43:48.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:43:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:43:47.692+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:43:48.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:43:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:43:47.693+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:43:48.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:43:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:43:47.702+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:43:48.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:43:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:43:47.702+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:43:48.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:43:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:43:47.702+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:43:48.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:43:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:43:47.703+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:43:48.569 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:43:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:43:48.150+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:43:48.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:43:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:43:48.150+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:43:48.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:43:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:43:48.150+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:43:48.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:43:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:43:48.150+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:43:49.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:43:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:43:48.698+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:43:49.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:43:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:43:48.698+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:43:49.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:43:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:43:48.698+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:43:49.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:43:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:43:48.698+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:43:49.071 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:43:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:43:48.698+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:43:49.071 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:43:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:43:48.698+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:43:49.071 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:43:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:43:48.698+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:43:49.071 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:43:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:43:48.699+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:43:49.569 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:43:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:43:49.183+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:43:49.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:43:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:43:49.183+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:43:49.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:43:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:43:49.183+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:43:49.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:43:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:43:49.183+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:43:50.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:43:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:43:49.713+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:43:50.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:43:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:43:49.713+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:43:50.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:43:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:43:49.713+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:43:50.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:43:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:43:49.713+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:43:50.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:43:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:43:49.718+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:43:50.071 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:43:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:43:49.718+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:43:50.071 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:43:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:43:49.718+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:43:50.071 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:43:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:43:49.719+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:43:50.461 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:43:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:43:50.213+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:43:50.462 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:43:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:43:50.213+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:43:50.462 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:43:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:43:50.213+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:43:50.462 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:43:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:43:50.214+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:43:50.742 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:43:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:43:50.459+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 21007 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:43:50.742 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:43:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:43:50.708+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:43:50.742 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:43:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:43:50.708+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:43:50.743 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:43:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:43:50.708+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:43:50.743 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:43:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:43:50.709+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:43:51.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:43:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:43:50.741+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:43:51.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:43:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:43:50.741+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:43:51.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:43:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:43:50.741+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:43:51.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:43:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:43:50.741+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:43:51.569 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:43:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:43:51.244+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:43:51.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:43:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:43:51.244+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:43:51.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:43:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:43:51.244+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:43:51.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:43:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:43:51.244+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:43:52.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:43:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:43:51.731+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:43:52.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:43:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:43:51.731+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:43:52.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:43:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:43:51.731+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:43:52.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:43:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:43:51.732+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:43:52.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:43:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:43:51.696+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:43:52.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:43:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:43:51.696+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:43:52.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:43:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:43:51.696+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:43:52.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:43:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:43:51.697+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:43:52.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:43:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:43:52.286+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:43:52.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:43:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:43:52.286+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:43:52.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:43:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:43:52.286+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:43:52.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:43:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:43:52.287+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:43:53.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:43:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:43:52.757+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:43:53.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:43:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:43:52.757+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:43:53.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:43:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:43:52.757+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:43:53.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:43:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:43:52.758+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:43:53.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:43:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:43:52.682+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:43:53.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:43:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:43:52.682+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:43:53.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:43:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:43:52.682+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:43:53.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:43:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:43:52.683+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:43:53.569 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:43:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:43:53.308+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:43:53.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:43:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:43:53.308+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:43:53.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:43:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:43:53.308+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:43:53.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:43:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:43:53.309+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:43:54.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:43:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:43:53.765+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:43:54.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:43:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:43:53.765+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:43:54.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:43:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:43:53.765+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:43:54.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:43:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:43:53.766+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:43:54.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:43:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:43:53.654+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:43:54.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:43:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:43:53.654+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:43:54.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:43:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:43:53.654+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:43:54.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:43:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:43:53.654+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:43:54.569 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:43:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:43:54.282+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:43:54.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:43:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:43:54.282+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:43:54.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:43:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:43:54.282+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:43:54.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:43:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:43:54.282+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:43:55.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:43:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:43:54.755+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:43:55.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:43:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:43:54.755+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:43:55.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:43:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:43:54.755+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:43:55.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:43:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:43:54.755+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:43:55.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:43:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:43:54.659+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:43:55.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:43:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:43:54.659+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:43:55.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:43:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:43:54.659+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:43:55.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:43:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:43:54.660+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:43:55.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:43:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:43:55.461+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 21091 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:43:55.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:43:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:43:55.261+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:43:55.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:43:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:43:55.261+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:43:55.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:43:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:43:55.261+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:43:55.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:43:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:43:55.262+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:43:56.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:43:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:43:55.795+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:43:56.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:43:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:43:55.795+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:43:56.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:43:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:43:55.795+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:43:56.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:43:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:43:55.796+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:43:56.073 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:43:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:43:55.654+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:43:56.073 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:43:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:43:55.654+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:43:56.073 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:43:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:43:55.654+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:43:56.073 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:43:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:43:55.655+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:43:56.569 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:43:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:43:56.243+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:43:56.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:43:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:43:56.243+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:43:56.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:43:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:43:56.243+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:43:56.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:43:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:43:56.243+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:43:57.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:43:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:43:56.649+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:43:57.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:43:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:43:56.649+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:43:57.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:43:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:43:56.649+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:43:57.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:43:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:43:56.649+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:43:57.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:43:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:43:56.839+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:43:57.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:43:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:43:56.839+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:43:57.071 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:43:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:43:56.839+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:43:57.071 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:43:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:43:56.839+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:43:57.569 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:43:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:43:57.251+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:43:57.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:43:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:43:57.251+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:43:57.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:43:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:43:57.251+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:43:57.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:43:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:43:57.251+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:43:58.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:43:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:43:57.681+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:43:58.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:43:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:43:57.681+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:43:58.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:43:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:43:57.681+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:43:58.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:43:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:43:57.682+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:43:58.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:43:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:43:57.857+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:43:58.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:43:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:43:57.857+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:43:58.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:43:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:43:57.857+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:43:58.071 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:43:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:43:57.857+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:43:58.569 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:43:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:43:58.239+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:43:58.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:43:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:43:58.239+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:43:58.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:43:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:43:58.239+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:43:58.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:43:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:43:58.239+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:43:59.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:43:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:43:58.718+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:43:59.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:43:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:43:58.718+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:43:59.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:43:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:43:58.718+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:43:59.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:43:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:43:58.718+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:43:59.071 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:43:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:43:58.825+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:43:59.071 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:43:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:43:58.825+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:43:59.071 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:43:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:43:58.825+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:43:59.072 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:43:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:43:58.825+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:43:59.569 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:43:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:43:59.234+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:43:59.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:43:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:43:59.234+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:43:59.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:43:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:43:59.234+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:43:59.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:43:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:43:59.234+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:44:00.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:43:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:43:59.798+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:44:00.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:43:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:43:59.798+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:44:00.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:43:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:43:59.798+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:44:00.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:43:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:43:59.799+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:44:00.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:43:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:43:59.739+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:44:00.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:43:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:43:59.739+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:44:00.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:43:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:43:59.739+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:44:00.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:43:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:43:59.740+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:44:00.465 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:44:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:44:00.267+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:44:00.465 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:44:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:44:00.267+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:44:00.465 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:44:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:44:00.267+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:44:00.465 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:44:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:44:00.268+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:44:00.752 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:44:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:44:00.464+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 21173 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:44:01.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:44:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:44:00.843+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:44:01.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:44:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:44:00.843+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:44:01.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:44:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:44:00.843+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:44:01.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:44:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:44:00.844+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:44:01.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:44:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:44:00.751+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:44:01.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:44:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:44:00.751+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:44:01.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:44:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:44:00.751+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:44:01.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:44:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:44:00.752+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:44:01.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:44:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:44:01.314+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:44:01.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:44:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:44:01.314+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:44:01.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:44:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:44:01.314+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:44:01.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:44:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:44:01.314+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:44:02.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:44:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:44:01.835+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:44:02.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:44:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:44:01.835+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:44:02.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:44:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:44:01.835+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:44:02.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:44:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:44:01.836+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:44:02.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:44:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:44:01.738+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:44:02.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:44:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:44:01.738+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:44:02.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:44:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:44:01.738+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:44:02.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:44:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:44:01.739+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:44:02.569 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:44:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:44:02.296+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:44:02.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:44:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:44:02.296+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:44:02.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:44:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:44:02.296+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:44:02.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:44:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:44:02.296+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:44:03.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:44:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:44:02.780+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:44:03.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:44:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:44:02.780+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:44:03.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:44:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:44:02.780+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:44:03.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:44:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:44:02.781+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:44:03.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:44:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:44:02.841+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:44:03.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:44:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:44:02.841+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:44:03.071 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:44:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:44:02.841+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:44:03.071 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:44:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:44:02.842+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:44:03.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:44:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:44:03.269+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:44:03.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:44:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:44:03.269+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:44:03.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:44:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:44:03.269+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:44:03.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:44:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:44:03.269+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:44:04.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:44:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:44:03.815+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:44:04.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:44:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:44:03.815+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:44:04.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:44:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:44:03.815+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:44:04.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:44:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:44:03.816+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:44:04.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:44:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:44:03.885+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:44:04.071 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:44:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:44:03.885+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:44:04.071 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:44:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:44:03.885+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:44:04.071 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:44:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:44:03.886+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:44:04.571 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:44:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:44:04.230+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:44:04.571 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:44:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:44:04.230+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:44:04.571 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:44:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:44:04.230+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:44:04.571 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:44:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:44:04.230+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:44:05.243 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:44:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:44:04.847+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:44:05.243 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:44:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:44:04.847+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:44:05.244 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:44:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:44:04.847+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:44:05.244 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:44:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:44:04.847+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:44:05.244 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:44:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:44:04.858+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:44:05.244 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:44:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:44:04.858+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:44:05.244 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:44:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:44:04.858+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:44:05.244 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:44:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:44:04.859+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:44:05.569 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:44:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:44:05.466+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 21257 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:44:05.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:44:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:44:05.242+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:44:05.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:44:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:44:05.242+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:44:05.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:44:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:44:05.242+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:44:05.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:44:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:44:05.242+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:44:05.842 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:44:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:44:05.815+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:44:05.842 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:44:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:44:05.815+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:44:05.842 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:44:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:44:05.815+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:44:05.842 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:44:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:44:05.816+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:44:06.282 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:44:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:44:05.840+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:44:06.282 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:44:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:44:05.840+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:44:06.282 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:44:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:44:05.840+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:44:06.282 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:44:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:44:05.841+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:44:06.569 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:44:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:44:06.280+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:44:06.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:44:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:44:06.280+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:44:06.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:44:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:44:06.280+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:44:06.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:44:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:44:06.281+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:44:07.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:44:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:44:06.846+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:44:07.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:44:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:44:06.846+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:44:07.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:44:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:44:06.846+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:44:07.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:44:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:44:06.847+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:44:07.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:44:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:44:06.779+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:44:07.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:44:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:44:06.779+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:44:07.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:44:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:44:06.779+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:44:07.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:44:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:44:06.779+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:44:07.569 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:44:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:44:07.304+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:44:07.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:44:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:44:07.304+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:44:07.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:44:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:44:07.304+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:44:07.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:44:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:44:07.304+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:44:08.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:44:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:44:07.862+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:44:08.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:44:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:44:07.862+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:44:08.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:44:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:44:07.862+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:44:08.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:44:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:44:07.862+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:44:08.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:44:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:44:07.825+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:44:08.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:44:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:44:07.825+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:44:08.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:44:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:44:07.825+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:44:08.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:44:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:44:07.825+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:44:08.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:44:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:44:08.345+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:44:08.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:44:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:44:08.345+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:44:08.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:44:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:44:08.345+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:44:08.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:44:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:44:08.345+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:44:09.116 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:44:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:44:08.882+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:44:09.116 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:44:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:44:08.882+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:44:09.116 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:44:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:44:08.882+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:44:09.117 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:44:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:44:08.884+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:44:09.117 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:44:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:44:08.843+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:44:09.117 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:44:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:44:08.843+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:44:09.117 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:44:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:44:08.843+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:44:09.118 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:44:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:44:08.845+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:44:09.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:44:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:44:09.307+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:44:09.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:44:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:44:09.307+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:44:09.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:44:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:44:09.307+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:44:09.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:44:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:44:09.307+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:44:10.182 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:44:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:44:09.884+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:44:10.183 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:44:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:44:09.884+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:44:10.183 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:44:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:44:09.884+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:44:10.183 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:44:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:44:09.885+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:44:10.184 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:44:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:44:09.876+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:44:10.184 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:44:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:44:09.876+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:44:10.185 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:44:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:44:09.876+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:44:10.185 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:44:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:44:09.877+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:44:10.470 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:44:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:44:10.266+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:44:10.471 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:44:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:44:10.266+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:44:10.471 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:44:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:44:10.266+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:44:10.471 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:44:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:44:10.267+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:44:10.819 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:44:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:44:10.468+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 21339 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:44:11.277 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:44:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:44:10.848+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:44:11.277 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:44:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:44:10.848+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:44:11.277 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:44:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:44:10.848+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:44:11.278 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:44:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:44:10.848+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:44:11.278 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:44:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:44:10.900+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:44:11.278 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:44:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:44:10.900+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:44:11.278 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:44:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:44:10.900+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:44:11.278 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:44:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:44:10.900+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:44:11.569 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:44:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:44:11.276+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:44:11.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:44:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:44:11.276+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:44:11.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:44:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:44:11.276+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:44:11.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:44:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:44:11.276+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:44:12.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:44:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:44:11.910+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:44:12.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:44:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:44:11.910+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:44:12.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:44:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:44:11.910+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:44:12.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:44:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:44:11.911+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:44:12.071 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:44:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:44:11.851+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:44:12.071 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:44:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:44:11.851+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:44:12.071 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:44:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:44:11.851+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:44:12.071 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:44:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:44:11.852+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:44:12.569 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:44:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:44:12.269+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:44:12.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:44:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:44:12.269+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:44:12.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:44:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:44:12.269+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:44:12.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:44:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:44:12.270+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:44:13.287 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:44:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:44:12.911+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:44:13.287 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:44:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:44:12.911+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:44:13.287 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:44:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:44:12.911+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:44:13.287 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:44:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:44:12.912+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:44:13.288 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:44:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:44:12.854+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:44:13.288 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:44:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:44:12.854+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:44:13.288 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:44:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:44:12.854+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:44:13.288 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:44:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:44:12.855+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:44:13.569 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:44:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:44:13.285+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:44:13.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:44:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:44:13.285+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:44:13.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:44:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:44:13.285+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:44:13.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:44:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:44:13.286+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:44:14.298 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:44:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:44:13.860+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:44:14.298 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:44:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:44:13.860+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:44:14.298 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:44:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:44:13.860+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:44:14.298 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:44:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:44:13.861+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:44:14.299 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:44:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:44:13.941+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:44:14.299 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:44:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:44:13.941+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:44:14.299 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:44:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:44:13.941+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:44:14.299 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:44:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:44:13.941+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:44:14.569 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:44:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:44:14.297+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:44:14.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:44:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:44:14.297+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:44:14.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:44:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:44:14.297+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:44:14.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:44:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:44:14.297+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:44:15.319 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:44:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:44:14.962+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:44:15.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:44:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:44:14.962+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:44:15.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:44:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:44:14.962+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:44:15.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:44:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:44:14.962+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:44:15.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:44:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:44:14.872+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:44:15.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:44:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:44:14.872+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:44:15.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:44:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:44:14.872+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:44:15.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:44:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:44:14.873+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:44:15.819 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:44:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:44:15.470+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 21423 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:44:15.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:44:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:44:15.344+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:44:15.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:44:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:44:15.344+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:44:15.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:44:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:44:15.344+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:44:15.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:44:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:44:15.344+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:44:16.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:44:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:44:15.832+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:44:16.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:44:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:44:15.832+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:44:16.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:44:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:44:15.832+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:44:16.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:44:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:44:15.833+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:44:16.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:44:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:44:15.912+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:44:16.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:44:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:44:15.912+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:44:16.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:44:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:44:15.912+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:44:16.321 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:44:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:44:15.913+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:44:16.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:44:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:44:16.331+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:44:16.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:44:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:44:16.331+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:44:16.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:44:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:44:16.331+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:44:16.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:44:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:44:16.332+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:44:17.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:44:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:44:16.844+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:44:17.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:44:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:44:16.844+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:44:17.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:44:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:44:16.844+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:44:17.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:44:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:44:16.845+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:44:17.321 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:44:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:44:16.889+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:44:17.321 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:44:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:44:16.889+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:44:17.321 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:44:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:44:16.889+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:44:17.321 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:44:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:44:16.889+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:44:17.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:44:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:44:17.345+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:44:17.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:44:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:44:17.345+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:44:17.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:44:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:44:17.345+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:44:17.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:44:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:44:17.345+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:44:18.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:44:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:44:17.890+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:44:18.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:44:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:44:17.890+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:44:18.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:44:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:44:17.890+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:44:18.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:44:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:44:17.891+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:44:18.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:44:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:44:17.903+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:44:18.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:44:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:44:17.903+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:44:18.321 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:44:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:44:17.903+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:44:18.321 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:44:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:44:17.903+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:44:18.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:44:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:44:18.365+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:44:18.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:44:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:44:18.365+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:44:18.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:44:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:44:18.365+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:44:18.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:44:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:44:18.366+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:44:19.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:44:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:44:18.921+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:44:19.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:44:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:44:18.921+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:44:19.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:44:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:44:18.921+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:44:19.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:44:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:44:18.922+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:44:19.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:44:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:44:18.940+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:44:19.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:44:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:44:18.940+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:44:19.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:44:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:44:18.940+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:44:19.321 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:44:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:44:18.941+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:44:19.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:44:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:44:19.354+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:44:19.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:44:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:44:19.354+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:44:19.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:44:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:44:19.354+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:44:19.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:44:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:44:19.354+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:44:20.183 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:44:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:44:19.978+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:44:20.183 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:44:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:44:19.978+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:44:20.183 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:44:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:44:19.978+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:44:20.183 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:44:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:44:19.979+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:44:20.183 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:44:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:44:19.930+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:44:20.184 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:44:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:44:19.930+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:44:20.184 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:44:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:44:19.930+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:44:20.184 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:44:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:44:19.930+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:44:20.474 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:44:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:44:20.366+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:44:20.474 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:44:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:44:20.366+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:44:20.474 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:44:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:44:20.366+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:44:20.474 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:44:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:44:20.367+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:44:20.819 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:44:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:44:20.472+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 21505 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:44:21.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:44:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:44:20.974+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:44:21.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:44:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:44:20.974+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:44:21.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:44:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:44:20.974+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:44:21.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:44:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:44:20.974+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:44:21.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:44:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:44:21.028+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:44:21.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:44:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:44:21.028+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:44:21.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:44:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:44:21.028+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:44:21.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:44:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:44:21.028+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:44:21.715 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:44:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:44:21.328+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:44:21.715 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:44:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:44:21.328+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:44:21.715 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:44:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:44:21.328+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:44:21.715 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:44:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:44:21.329+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:44:22.301 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:44:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:44:21.996+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:44:22.301 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:44:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:44:21.996+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:44:22.301 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:44:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:44:21.996+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:44:22.302 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:44:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:44:21.996+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:44:22.302 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:44:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:44:21.979+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:44:22.302 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:44:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:44:21.979+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:44:22.302 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:44:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:44:21.979+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:44:22.302 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:44:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:44:21.979+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:44:22.569 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:44:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:44:22.300+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:44:22.569 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:44:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:44:22.300+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:44:22.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:44:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:44:22.300+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:44:22.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:44:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:44:22.300+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:44:23.285 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:44:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:44:23.002+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:44:23.285 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:44:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:44:23.002+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:44:23.285 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:44:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:44:23.002+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:44:23.285 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:44:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:44:23.002+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:44:23.286 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:44:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:44:22.963+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:44:23.286 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:44:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:44:22.963+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:44:23.286 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:44:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:44:22.963+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:44:23.286 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:44:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:44:22.964+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:44:23.569 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:44:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:44:23.284+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:44:23.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:44:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:44:23.284+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:44:23.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:44:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:44:23.284+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:44:23.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:44:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:44:23.284+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:44:24.247 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:44:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:44:23.998+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:44:24.248 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:44:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:44:23.998+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:44:24.248 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:44:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:44:23.998+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:44:24.248 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:44:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:44:23.999+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:44:24.248 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:44:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:44:23.974+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:44:24.248 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:44:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:44:23.974+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:44:24.248 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:44:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:44:23.974+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:44:24.249 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:44:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:44:23.975+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:44:24.569 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:44:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:44:24.246+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:44:24.569 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:44:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:44:24.246+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:44:24.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:44:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:44:24.246+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:44:24.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:44:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:44:24.247+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:44:25.319 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:44:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:44:25.008+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:44:25.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:44:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:44:25.008+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:44:25.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:44:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:44:25.008+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:44:25.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:44:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:44:25.008+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:44:25.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:44:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:44:24.981+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:44:25.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:44:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:44:24.981+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:44:25.321 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:44:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:44:24.981+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:44:25.321 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:44:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:44:24.981+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:44:25.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:44:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:44:25.220+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:44:25.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:44:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:44:25.220+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:44:25.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:44:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:44:25.220+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:44:25.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:44:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:44:25.220+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:44:25.819 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:44:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:44:25.474+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 21589 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:44:26.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:44:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:44:25.956+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:44:26.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:44:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:44:25.956+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:44:26.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:44:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:44:25.956+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:44:26.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:44:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:44:25.957+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:44:26.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:44:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:44:26.034+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:44:26.321 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:44:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:44:26.034+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:44:26.321 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:44:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:44:26.034+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:44:26.321 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:44:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:44:26.035+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:44:26.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:44:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:44:26.194+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:44:26.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:44:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:44:26.194+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:44:26.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:44:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:44:26.194+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:44:26.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:44:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:44:26.195+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:44:27.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:44:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:44:27.164+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:44:27.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:44:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:44:27.164+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:44:27.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:44:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:44:27.164+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:44:27.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:44:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:44:27.165+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:44:27.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:44:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:44:26.996+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:44:27.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:44:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:44:26.996+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:44:27.321 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:44:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:44:26.996+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:44:27.321 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:44:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:44:26.996+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:44:27.321 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:44:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:44:27.036+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:44:27.321 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:44:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:44:27.036+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:44:27.321 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:44:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:44:27.036+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:44:27.321 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:44:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:44:27.036+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:44:28.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:44:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:44:27.980+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:44:28.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:44:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:44:27.980+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:44:28.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:44:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:44:27.980+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:44:28.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:44:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:44:27.981+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:44:28.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:44:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:44:28.200+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:44:28.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:44:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:44:28.200+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:44:28.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:44:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:44:28.200+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:44:28.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:44:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:44:28.200+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:44:28.321 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:44:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:44:28.000+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:44:28.321 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:44:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:44:28.000+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:44:28.321 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:44:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:44:28.000+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:44:28.321 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:44:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:44:28.000+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:44:29.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:44:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:44:29.207+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:44:29.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:44:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:44:29.207+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:44:29.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:44:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:44:29.207+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:44:29.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:44:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:44:29.207+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:44:29.321 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:44:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:44:28.993+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:44:29.321 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:44:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:44:28.993+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:44:29.321 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:44:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:44:28.993+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:44:29.321 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:44:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:44:28.994+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:44:29.321 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:44:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:44:28.982+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:44:29.321 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:44:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:44:28.982+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:44:29.321 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:44:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:44:28.982+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:44:29.322 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:44:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:44:28.983+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:44:30.244 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:44:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:44:29.959+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:44:30.244 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:44:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:44:29.959+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:44:30.244 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:44:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:44:29.959+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:44:30.244 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:44:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:44:29.959+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:44:30.244 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:44:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:44:30.005+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:44:30.245 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:44:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:44:30.005+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:44:30.245 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:44:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:44:30.005+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:44:30.245 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:44:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:44:30.005+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:44:30.503 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:44:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:44:30.476+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 21671 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:44:30.503 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:44:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:44:30.242+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:44:30.503 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:44:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:44:30.242+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:44:30.503 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:44:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:44:30.242+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:44:30.504 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:44:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:44:30.243+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:44:31.236 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:44:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:44:30.960+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:44:31.236 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:44:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:44:30.960+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:44:31.236 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:44:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:44:30.960+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:44:31.236 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:44:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:44:30.961+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:44:31.237 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:44:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:44:30.929+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:44:31.237 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:44:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:44:30.929+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:44:31.237 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:44:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:44:30.929+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:44:31.237 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:44:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:44:30.930+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:44:31.569 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:44:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:44:31.234+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:44:31.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:44:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:44:31.234+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:44:31.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:44:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:44:31.234+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:44:31.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:44:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:44:31.235+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:44:32.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:44:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:44:31.895+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:44:32.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:44:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:44:31.895+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:44:32.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:44:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:44:31.895+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:44:32.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:44:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:44:31.895+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:44:32.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:44:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:44:31.913+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:44:32.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:44:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:44:31.913+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:44:32.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:44:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:44:31.913+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:44:32.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:44:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:44:31.914+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:44:32.569 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:44:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:44:32.235+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:44:32.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:44:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:44:32.235+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:44:32.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:44:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:44:32.235+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:44:32.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:44:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:44:32.235+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:44:33.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:44:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:44:32.884+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:44:33.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:44:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:44:32.884+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:44:33.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:44:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:44:32.884+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:44:33.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:44:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:44:32.885+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:44:33.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:44:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:44:32.940+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:44:33.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:44:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:44:32.940+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:44:33.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:44:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:44:32.940+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:44:33.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:44:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:44:32.941+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:44:33.569 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:44:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:44:33.198+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:44:33.569 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:44:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:44:33.198+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:44:33.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:44:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:44:33.198+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:44:33.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:44:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:44:33.198+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:44:34.197 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:44:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:44:33.963+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:44:34.197 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:44:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:44:33.963+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:44:34.197 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:44:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:44:33.963+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:44:34.197 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:44:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:44:33.964+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:44:34.197 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:44:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:44:33.879+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:44:34.197 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:44:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:44:33.879+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:44:34.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:44:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:44:33.879+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:44:34.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:44:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:44:33.880+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:44:34.569 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:44:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:44:34.195+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:44:34.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:44:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:44:34.195+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:44:34.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:44:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:44:34.195+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:44:34.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:44:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:44:34.196+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:44:35.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:44:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:44:34.830+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:44:35.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:44:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:44:34.830+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:44:35.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:44:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:44:34.830+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:44:35.197 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:44:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:44:34.831+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:44:35.197 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:44:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:44:34.949+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:44:35.197 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:44:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:44:34.949+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:44:35.197 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:44:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:44:34.949+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:44:35.197 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:44:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:44:34.949+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:44:35.481 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:44:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:44:35.194+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:44:35.481 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:44:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:44:35.194+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:44:35.481 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:44:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:44:35.194+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:44:35.481 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:44:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:44:35.194+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:44:35.800 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:44:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:44:35.478+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 21754 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:44:36.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:44:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:44:35.910+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:44:36.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:44:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:44:35.910+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:44:36.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:44:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:44:35.910+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:44:36.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:44:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:44:35.911+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:44:36.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:44:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:44:35.798+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:44:36.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:44:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:44:35.798+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:44:36.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:44:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:44:35.798+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:44:36.071 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:44:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:44:35.799+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:44:36.569 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:44:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:44:36.189+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:44:36.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:44:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:44:36.189+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:44:36.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:44:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:44:36.189+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:44:36.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:44:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:44:36.189+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:44:37.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:44:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:44:36.770+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:44:37.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:44:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:44:36.770+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:44:37.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:44:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:44:36.770+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:44:37.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:44:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:44:36.771+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:44:37.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:44:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:44:36.916+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:44:37.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:44:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:44:36.916+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:44:37.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:44:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:44:36.916+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:44:37.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:44:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:44:36.917+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:44:37.569 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:44:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:44:37.144+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:44:37.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:44:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:44:37.144+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:44:37.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:44:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:44:37.144+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:44:37.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:44:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:44:37.144+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:44:38.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:44:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:44:37.809+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:44:38.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:44:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:44:37.809+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:44:38.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:44:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:44:37.809+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:44:38.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:44:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:44:37.810+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:44:38.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:44:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:44:37.965+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:44:38.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:44:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:44:37.965+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:44:38.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:44:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:44:37.965+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:44:38.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:44:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:44:37.966+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:44:38.569 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:44:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:44:38.109+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:44:38.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:44:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:44:38.109+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:44:38.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:44:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:44:38.109+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:44:38.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:44:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:44:38.110+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:44:39.115 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:44:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:44:38.854+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:44:39.116 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:44:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:44:38.854+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:44:39.116 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:44:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:44:38.854+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:44:39.116 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:44:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:44:38.855+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:44:39.116 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:44:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:44:38.934+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:44:39.116 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:44:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:44:38.934+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:44:39.117 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:44:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:44:38.934+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:44:39.117 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:44:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:44:38.935+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:44:39.117 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:44:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:44:39.083+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:44:39.117 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:44:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:44:39.083+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:44:39.117 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:44:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:44:39.083+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:44:39.117 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:44:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:44:39.084+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:44:40.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:44:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:44:39.950+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:44:40.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:44:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:44:39.950+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:44:40.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:44:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:44:39.950+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:44:40.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:44:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:44:39.950+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:44:40.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:44:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:44:39.811+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:44:40.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:44:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:44:39.811+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:44:40.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:44:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:44:39.811+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:44:40.071 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:44:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:44:39.811+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:44:40.482 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:44:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:44:40.087+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:44:40.483 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:44:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:44:40.087+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:44:40.483 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:44:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:44:40.087+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:44:40.483 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:44:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:44:40.088+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:44:40.819 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:44:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:44:40.480+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 21837 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:44:41.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:44:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:44:41.081+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:44:41.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:44:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:44:41.081+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:44:41.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:44:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:44:41.081+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:44:41.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:44:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:44:41.081+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:44:41.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:44:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:44:40.840+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:44:41.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:44:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:44:40.840+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:44:41.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:44:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:44:40.840+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:44:41.321 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:44:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:44:40.841+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:44:41.321 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:44:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:44:40.911+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:44:41.321 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:44:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:44:40.911+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:44:41.321 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:44:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:44:40.911+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:44:41.321 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:44:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:44:40.912+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:44:42.011 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:44:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:44:41.929+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:44:42.011 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:44:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:44:41.929+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:44:42.012 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:44:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:44:41.929+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:44:42.012 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:44:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:44:41.930+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:44:42.012 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:44:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:44:41.872+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:44:42.012 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:44:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:44:41.872+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:44:42.012 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:44:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:44:41.872+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:44:42.012 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:44:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:44:41.872+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:44:42.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:44:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:44:42.082+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:44:42.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:44:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:44:42.082+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:44:42.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:44:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:44:42.082+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:44:42.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:44:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:44:42.083+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:44:43.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:44:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:44:42.904+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:44:43.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:44:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:44:42.904+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:44:43.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:44:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:44:42.904+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:44:43.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:44:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:44:42.904+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:44:43.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:44:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:44:42.849+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:44:43.321 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:44:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:44:42.849+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:44:43.321 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:44:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:44:42.849+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:44:43.321 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:44:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:44:42.850+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:44:43.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:44:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:44:43.046+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:44:43.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:44:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:44:43.046+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:44:43.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:44:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:44:43.046+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:44:43.322 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:44:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:44:43.047+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:44:44.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:44:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:44:44.005+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:44:44.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:44:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:44:44.005+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:44:44.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:44:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:44:44.005+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:44:44.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:44:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:44:44.006+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:44:44.222 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:44:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:44:43.861+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:44:44.222 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:44:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:44:43.861+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:44:44.222 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:44:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:44:43.861+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:44:44.222 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:44:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:44:43.862+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:44:44.222 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:44:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:44:43.914+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:44:44.222 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:44:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:44:43.914+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:44:44.223 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:44:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:44:43.914+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:44:44.223 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:44:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:44:43.914+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:44:45.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:44:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:44:44.984+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:44:45.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:44:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:44:44.984+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:44:45.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:44:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:44:44.984+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:44:45.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:44:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:44:44.985+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:44:45.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:44:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:44:44.834+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:44:45.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:44:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:44:44.834+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:44:45.321 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:44:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:44:44.834+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:44:45.321 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:44:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:44:44.834+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:44:45.321 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:44:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:44:44.942+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:44:45.321 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:44:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:44:44.942+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:44:45.321 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:44:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:44:44.942+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:44:45.321 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:44:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:44:44.943+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:44:45.819 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:44:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:44:45.483+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 21920 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:44:46.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:44:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:44:45.819+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:44:46.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:44:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:44:45.819+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:44:46.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:44:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:44:45.819+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:44:46.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:44:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:44:45.820+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:44:46.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:44:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:44:45.983+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:44:46.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:44:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:44:45.983+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:44:46.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:44:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:44:45.983+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:44:46.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:44:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:44:45.983+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:44:46.321 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:44:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:44:45.899+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:44:46.321 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:44:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:44:45.899+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:44:46.321 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:44:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:44:45.899+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:44:46.321 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:44:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:44:45.900+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:44:47.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:44:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:44:46.807+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:44:47.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:44:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:44:46.807+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:44:47.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:44:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:44:46.808+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:44:47.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:44:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:44:46.808+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:44:47.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:44:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:44:47.022+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:44:47.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:44:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:44:47.022+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:44:47.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:44:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:44:47.022+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:44:47.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:44:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:44:47.022+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:44:47.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:44:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:44:46.934+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:44:47.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:44:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:44:46.934+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:44:47.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:44:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:44:46.934+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:44:47.072 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:44:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:44:46.934+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:44:48.053 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:44:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:44:47.906+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:44:48.054 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:44:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:44:47.906+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:44:48.054 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:44:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:44:47.906+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:44:48.054 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:44:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:44:47.907+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:44:48.054 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:44:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:44:47.766+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:44:48.054 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:44:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:44:47.766+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:44:48.055 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:44:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:44:47.766+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:44:48.055 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:44:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:44:47.767+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:44:48.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:44:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:44:48.052+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:44:48.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:44:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:44:48.052+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:44:48.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:44:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:44:48.052+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:44:48.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:44:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:44:48.053+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:44:49.031 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:44:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:44:48.724+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:44:49.031 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:44:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:44:48.724+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:44:49.031 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:44:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:44:48.724+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:44:49.031 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:44:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:44:48.725+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:44:49.032 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:44:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:44:48.916+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:44:49.032 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:44:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:44:48.916+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:44:49.032 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:44:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:44:48.916+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:44:49.032 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:44:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:44:48.916+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:44:49.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:44:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:44:49.029+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:44:49.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:44:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:44:49.029+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:44:49.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:44:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:44:49.029+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:44:49.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:44:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:44:49.030+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:44:50.049 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:44:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:44:49.700+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:44:50.049 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:44:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:44:49.700+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:44:50.049 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:44:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:44:49.700+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:44:50.050 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:44:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:44:49.701+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:44:50.050 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:44:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:44:49.941+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:44:50.050 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:44:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:44:49.941+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:44:50.050 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:44:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:44:49.941+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:44:50.050 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:44:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:44:49.942+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:44:50.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:44:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:44:50.047+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:44:50.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:44:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:44:50.047+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:44:50.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:44:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:44:50.047+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:44:50.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:44:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:44:50.048+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:44:50.741 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:44:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:44:50.485+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 22003 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:44:51.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:44:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:44:50.740+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:44:51.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:44:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:44:50.740+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:44:51.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:44:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:44:50.740+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:44:51.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:44:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:44:50.741+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:44:51.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:44:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:44:50.951+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:44:51.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:44:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:44:50.951+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:44:51.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:44:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:44:50.951+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:44:51.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:44:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:44:50.951+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:44:51.569 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:44:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:44:51.080+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:44:51.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:44:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:44:51.080+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:44:51.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:44:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:44:51.080+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:44:51.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:44:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:44:51.081+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:44:51.995 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:44:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:44:51.758+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:44:51.996 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:44:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:44:51.758+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:44:51.996 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:44:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:44:51.758+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:44:51.996 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:44:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:44:51.759+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:44:52.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:44:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:44:51.994+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:44:52.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:44:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:44:51.994+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:44:52.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:44:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:44:51.994+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:44:52.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:44:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:44:51.995+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:44:52.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:44:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:44:52.100+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:44:52.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:44:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:44:52.100+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:44:52.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:44:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:44:52.100+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:44:52.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:44:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:44:52.100+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:44:53.054 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:44:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:44:53.022+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:44:53.054 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:44:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:44:53.022+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:44:53.055 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:44:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:44:53.022+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:44:53.055 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:44:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:44:53.023+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:44:53.055 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:44:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:44:52.799+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:44:53.055 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:44:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:44:52.799+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:44:53.055 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:44:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:44:52.799+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:44:53.055 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:44:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:44:52.799+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:44:53.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:44:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:44:53.053+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:44:53.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:44:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:44:53.053+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:44:53.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:44:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:44:53.053+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:44:53.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:44:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:44:53.053+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:44:54.299 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:44:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:44:54.025+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:44:54.299 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:44:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:44:54.025+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:44:54.299 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:44:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:44:54.025+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:44:54.300 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:44:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:44:54.026+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:44:54.300 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:44:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:44:53.833+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:44:54.300 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:44:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:44:53.833+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:44:54.300 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:44:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:44:53.833+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:44:54.300 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:44:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:44:53.834+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:44:54.301 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:44:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:44:54.021+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:44:54.301 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:44:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:44:54.021+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:44:54.301 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:44:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:44:54.021+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:44:54.301 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:44:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:44:54.021+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:44:55.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:44:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:44:54.977+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:44:55.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:44:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:44:54.977+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:44:55.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:44:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:44:54.977+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:44:55.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:44:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:44:54.978+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:44:55.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:44:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:44:54.843+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:44:55.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:44:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:44:54.843+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:44:55.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:44:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:44:54.843+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:44:55.321 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:44:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:44:54.844+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:44:55.321 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:44:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:44:54.979+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:44:55.321 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:44:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:44:54.979+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:44:55.321 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:44:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:44:54.979+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:44:55.321 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:44:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:44:54.980+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:44:55.816 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:44:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:44:55.488+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 22086 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:44:56.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:44:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:44:55.815+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:44:56.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:44:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:44:55.815+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:44:56.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:44:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:44:55.815+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:44:56.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:44:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:44:55.816+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:44:56.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:44:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:44:56.024+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:44:56.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:44:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:44:56.024+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:44:56.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:44:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:44:56.024+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:44:56.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:44:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:44:56.025+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:44:56.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:44:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:44:55.989+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:44:56.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:44:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:44:55.989+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:44:56.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:44:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:44:55.989+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:44:56.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:44:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:44:55.989+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:44:57.052 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:44:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:44:56.775+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:44:57.052 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:44:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:44:56.775+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:44:57.052 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:44:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:44:56.775+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:44:57.053 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:44:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:44:56.776+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:44:57.053 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:44:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:44:56.991+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:44:57.053 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:44:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:44:56.991+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:44:57.053 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:44:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:44:56.991+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:44:57.053 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:44:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:44:56.991+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:44:57.319 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:44:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:44:57.051+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:44:57.319 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:44:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:44:57.051+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:44:57.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:44:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:44:57.051+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:44:57.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:44:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:44:57.051+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:44:58.008 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:44:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:44:57.734+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:44:58.009 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:44:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:44:57.734+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:44:58.009 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:44:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:44:57.734+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:44:58.009 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:44:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:44:57.735+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:44:58.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:44:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:44:58.036+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:44:58.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:44:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:44:58.036+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:44:58.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:44:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:44:58.036+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:44:58.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:44:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:44:58.036+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:44:58.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:44:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:44:58.007+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:44:58.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:44:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:44:58.007+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:44:58.321 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:44:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:44:58.007+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:44:58.321 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:44:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:44:58.008+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:44:59.008 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:44:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:44:58.737+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:44:59.008 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:44:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:44:58.737+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:44:59.008 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:44:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:44:58.737+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:44:59.008 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:44:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:44:58.738+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:44:59.319 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:44:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:44:59.006+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:44:59.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:44:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:44:59.006+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:44:59.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:44:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:44:59.006+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:44:59.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:44:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:44:59.007+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:44:59.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:44:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:44:59.008+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:44:59.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:44:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:44:59.008+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:44:59.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:44:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:44:59.008+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:44:59.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:44:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:44:59.008+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:44:59.916 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:44:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:44:59.723+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:44:59.916 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:44:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:44:59.723+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:44:59.916 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:44:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:44:59.723+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:44:59.916 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:44:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:44:59.723+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:45:00.185 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:44:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:44:59.963+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:45:00.186 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:44:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:44:59.963+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:45:00.186 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:44:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:44:59.963+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:45:00.186 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:44:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:44:59.963+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:45:00.186 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:44:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:44:59.964+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:45:00.186 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:44:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:44:59.964+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:45:00.187 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:44:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:44:59.964+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:45:00.187 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:44:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:44:59.964+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:45:00.713 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:45:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:45:00.490+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 22169 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:45:00.995 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:45:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:45:00.954+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:45:00.996 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:45:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:45:00.954+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:45:00.996 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:45:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:45:00.954+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:45:00.996 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:45:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:45:00.954+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:45:00.996 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:45:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:45:00.711+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:45:00.996 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:45:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:45:00.711+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:45:00.996 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:45:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:45:00.711+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:45:00.996 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:45:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:45:00.712+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:45:01.319 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:45:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:45:00.994+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:45:01.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:45:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:45:00.994+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:45:01.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:45:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:45:00.994+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:45:01.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:45:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:45:00.995+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:45:01.987 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:45:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:45:01.688+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:45:01.988 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:45:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:45:01.688+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:45:01.988 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:45:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:45:01.688+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:45:01.988 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:45:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:45:01.689+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:45:02.319 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:45:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:45:01.986+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:45:02.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:45:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:45:01.986+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:45:02.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:45:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:45:01.986+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:45:02.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:45:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:45:01.987+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:45:02.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:45:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:45:01.993+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:45:02.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:45:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:45:01.993+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:45:02.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:45:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:45:01.993+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:45:02.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:45:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:45:01.993+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:45:02.958 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:45:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:45:02.644+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:45:02.958 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:45:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:45:02.644+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:45:02.959 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:45:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:45:02.644+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:45:02.959 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:45:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:45:02.645+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:45:03.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:45:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:45:02.957+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:45:03.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:45:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:45:02.957+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:45:03.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:45:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:45:02.957+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:45:03.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:45:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:45:02.957+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:45:03.321 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:45:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:45:03.014+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:45:03.321 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:45:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:45:03.014+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:45:03.321 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:45:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:45:03.014+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:45:03.322 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:45:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:45:03.015+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:45:03.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:45:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:45:03.613+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:45:03.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:45:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:45:03.613+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:45:03.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:45:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:45:03.613+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:45:03.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:45:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:45:03.614+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:45:04.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:45:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:45:04.005+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:45:04.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:45:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:45:04.005+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:45:04.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:45:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:45:04.005+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:45:04.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:45:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:45:04.005+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:45:04.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:45:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:45:03.983+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:45:04.321 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:45:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:45:03.983+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:45:04.321 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:45:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:45:03.983+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:45:04.321 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:45:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:45:03.984+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:45:04.973 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:45:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:45:04.636+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:45:04.973 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:45:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:45:04.636+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:45:04.973 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:45:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:45:04.636+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:45:04.973 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:45:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:45:04.636+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:45:05.319 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:45:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:45:05.024+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:45:05.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:45:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:45:05.024+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:45:05.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:45:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:45:05.024+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:45:05.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:45:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:45:05.025+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:45:05.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:45:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:45:04.971+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:45:05.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:45:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:45:04.971+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:45:05.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:45:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:45:04.971+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:45:05.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:45:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:45:04.972+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:45:05.819 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:45:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:45:05.493+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 22252 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:45:05.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:45:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:45:05.662+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:45:05.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:45:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:45:05.662+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:45:05.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:45:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:45:05.662+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:45:05.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:45:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:45:05.662+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:45:06.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:45:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:45:06.003+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:45:06.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:45:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:45:06.003+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:45:06.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:45:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:45:06.003+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:45:06.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:45:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:45:06.004+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:45:06.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:45:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:45:05.929+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:45:06.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:45:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:45:05.929+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:45:06.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:45:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:45:05.929+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:45:06.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:45:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:45:05.929+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:45:06.959 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:45:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:45:06.668+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:45:06.959 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:45:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:45:06.668+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:45:06.959 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:45:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:45:06.668+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:45:06.959 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:45:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:45:06.668+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:45:07.319 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:45:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:45:07.009+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:45:07.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:45:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:45:07.009+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:45:07.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:45:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:45:07.009+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:45:07.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:45:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:45:07.009+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:45:07.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:45:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:45:06.958+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:45:07.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:45:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:45:06.958+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:45:07.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:45:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:45:06.958+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:45:07.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:45:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:45:06.958+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:45:07.958 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:45:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:45:07.634+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:45:07.958 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:45:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:45:07.634+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:45:07.959 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:45:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:45:07.634+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:45:07.959 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:45:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:45:07.635+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:45:08.319 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:45:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:45:08.047+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:45:08.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:45:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:45:08.047+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:45:08.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:45:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:45:08.047+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:45:08.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:45:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:45:08.049+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:45:08.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:45:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:45:07.957+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:45:08.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:45:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:45:07.957+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:45:08.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:45:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:45:07.957+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:45:08.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:45:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:45:07.957+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:45:08.774 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:45:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:45:08.685+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:45:08.774 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:45:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:45:08.685+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:45:08.775 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:45:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:45:08.685+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:45:08.775 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:45:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:45:08.685+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:45:09.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:45:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:45:08.916+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:45:09.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:45:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:45:08.916+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:45:09.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:45:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:45:08.916+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:45:09.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:45:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:45:08.917+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:45:09.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:45:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:45:09.077+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:45:09.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:45:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:45:09.077+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:45:09.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:45:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:45:09.077+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:45:09.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:45:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:45:09.077+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:45:10.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:45:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:45:09.868+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:45:10.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:45:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:45:09.868+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:45:10.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:45:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:45:09.868+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:45:10.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:45:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:45:09.868+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:45:10.071 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:45:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:45:09.713+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:45:10.071 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:45:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:45:09.713+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:45:10.071 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:45:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:45:09.713+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:45:10.071 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:45:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:45:09.713+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:45:10.496 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:45:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:45:10.068+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:45:10.497 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:45:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:45:10.068+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:45:10.497 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:45:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:45:10.068+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:45:10.497 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:45:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:45:10.069+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:45:10.819 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:45:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:45:10.495+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 22335 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:45:10.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:45:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:45:10.715+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:45:10.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:45:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:45:10.715+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:45:10.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:45:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:45:10.715+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:45:10.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:45:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:45:10.716+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:45:11.085 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:45:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:45:10.832+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:45:11.085 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:45:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:45:10.832+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:45:11.086 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:45:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:45:10.832+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:45:11.086 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:45:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:45:10.832+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:45:11.425 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:45:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:45:11.084+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:45:11.425 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:45:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:45:11.084+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:45:11.425 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:45:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:45:11.084+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:45:11.425 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:45:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:45:11.085+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:45:12.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:45:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:45:11.830+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:45:12.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:45:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:45:11.830+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:45:12.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:45:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:45:11.830+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:45:12.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:45:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:45:11.830+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:45:12.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:45:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:45:11.705+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:45:12.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:45:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:45:11.705+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:45:12.071 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:45:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:45:11.705+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:45:12.071 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:45:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:45:11.707+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:45:12.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:45:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:45:12.097+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:45:12.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:45:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:45:12.097+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:45:12.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:45:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:45:12.097+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:45:12.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:45:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:45:12.098+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:45:13.058 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:45:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:45:12.802+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:45:13.058 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:45:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:45:12.802+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:45:13.059 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:45:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:45:12.802+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:45:13.059 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:45:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:45:12.802+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:45:13.059 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:45:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:45:12.670+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:45:13.059 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:45:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:45:12.670+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:45:13.059 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:45:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:45:12.670+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:45:13.059 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:45:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:45:12.671+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:45:13.319 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:45:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:45:13.057+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:45:13.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:45:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:45:13.057+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:45:13.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:45:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:45:13.057+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:45:13.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:45:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:45:13.057+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:45:14.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:45:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:45:13.679+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:45:14.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:45:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:45:13.679+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:45:14.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:45:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:45:13.679+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:45:14.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:45:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:45:13.680+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:45:14.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:45:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:45:13.800+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:45:14.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:45:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:45:13.800+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:45:14.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:45:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:45:13.800+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:45:14.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:45:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:45:13.800+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:45:14.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:45:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:45:14.095+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:45:14.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:45:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:45:14.095+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:45:14.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:45:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:45:14.095+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:45:14.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:45:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:45:14.096+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:45:15.059 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:45:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:45:14.708+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:45:15.059 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:45:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:45:14.708+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:45:15.059 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:45:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:45:14.708+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:45:15.059 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:45:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:45:14.708+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:45:15.060 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:45:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:45:14.774+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:45:15.060 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:45:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:45:14.774+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:45:15.060 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:45:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:45:14.774+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:45:15.060 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:45:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:45:14.775+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:45:15.319 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:45:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:45:15.058+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:45:15.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:45:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:45:15.058+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:45:15.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:45:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:45:15.058+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:45:15.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:45:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:45:15.058+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:45:15.815 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:45:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:45:15.497+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 22418 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:45:15.816 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:45:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:45:15.662+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:45:15.816 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:45:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:45:15.662+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:45:15.816 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:45:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:45:15.662+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:45:15.816 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:45:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:45:15.662+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:45:16.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:45:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:45:16.062+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:45:16.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:45:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:45:16.062+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:45:16.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:45:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:45:16.062+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:45:16.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:45:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:45:16.063+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:45:16.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:45:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:45:15.814+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:45:16.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:45:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:45:15.814+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:45:16.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:45:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:45:15.814+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:45:16.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:45:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:45:15.814+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:45:17.038 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:45:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:45:16.801+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:45:17.038 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:45:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:45:16.801+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:45:17.038 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:45:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:45:16.801+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:45:17.038 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:45:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:45:16.802+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:45:17.038 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:45:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:45:16.654+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:45:17.039 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:45:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:45:16.654+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:45:17.039 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:45:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:45:16.654+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:45:17.039 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:45:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:45:16.654+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:45:17.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:45:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:45:17.036+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:45:17.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:45:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:45:17.036+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:45:17.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:45:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:45:17.036+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:45:17.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:45:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:45:17.037+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:45:18.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:45:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:45:17.635+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:45:18.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:45:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:45:17.635+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:45:18.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:45:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:45:17.635+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:45:18.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:45:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:45:17.636+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:45:18.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:45:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:45:17.814+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:45:18.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:45:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:45:17.814+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:45:18.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:45:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:45:17.814+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:45:18.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:45:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:45:17.815+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:45:18.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:45:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:45:18.072+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:45:18.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:45:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:45:18.072+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:45:18.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:45:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:45:18.072+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:45:18.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:45:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:45:18.072+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:45:19.051 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:45:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:45:18.839+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:45:19.052 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:45:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:45:18.839+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:45:19.052 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:45:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:45:18.839+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:45:19.052 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:45:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:45:18.840+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:45:19.052 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:45:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:45:18.681+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:45:19.052 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:45:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:45:18.681+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:45:19.052 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:45:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:45:18.681+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:45:19.053 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:45:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:45:18.682+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:45:19.319 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:45:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:45:19.050+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:45:19.319 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:45:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:45:19.050+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:45:19.319 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:45:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:45:19.050+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:45:19.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:45:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:45:19.051+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:45:19.826 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:45:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:45:19.668+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:45:19.826 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:45:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:45:19.668+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:45:19.826 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:45:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:45:19.668+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:45:19.827 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:45:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:45:19.669+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:45:20.086 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:45:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:45:19.825+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:45:20.087 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:45:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:45:19.825+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:45:20.087 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:45:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:45:19.825+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:45:20.087 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:45:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:45:19.825+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:45:20.501 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:45:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:45:20.085+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:45:20.501 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:45:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:45:20.085+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:45:20.501 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:45:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:45:20.085+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:45:20.502 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:45:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:45:20.086+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:45:20.819 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:45:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:45:20.500+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 22501 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:45:20.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:45:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:45:20.680+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:45:20.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:45:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:45:20.680+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:45:20.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:45:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:45:20.680+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:45:20.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:45:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:45:20.681+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:45:21.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:45:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:45:20.854+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:45:21.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:45:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:45:20.854+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:45:21.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:45:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:45:20.854+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:45:21.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:45:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:45:20.855+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:45:21.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:45:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:45:21.098+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:45:21.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:45:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:45:21.098+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:45:21.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:45:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:45:21.098+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:45:21.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:45:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:45:21.099+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:45:21.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:45:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:45:21.904+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:45:21.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:45:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:45:21.904+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:45:21.948 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:45:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:45:21.904+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:45:21.948 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:45:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:45:21.905+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:45:21.948 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:45:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:45:21.692+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:45:21.948 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:45:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:45:21.692+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:45:21.948 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:45:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:45:21.692+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:45:21.948 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:45:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:45:21.693+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:45:22.319 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:45:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:45:22.051+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:45:22.319 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:45:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:45:22.051+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:45:22.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:45:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:45:22.051+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:45:22.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:45:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:45:22.051+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:45:22.945 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:45:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:45:22.674+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:45:22.946 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:45:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:45:22.674+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:45:22.946 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:45:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:45:22.674+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:45:22.946 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:45:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:45:22.675+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:45:23.319 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:45:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:45:23.026+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:45:23.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:45:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:45:23.026+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:45:23.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:45:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:45:23.026+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:45:23.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:45:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:45:23.027+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:45:23.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:45:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:45:22.944+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:45:23.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:45:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:45:22.944+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:45:23.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:45:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:45:22.944+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:45:23.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:45:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:45:22.945+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:45:23.903 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:45:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:45:23.633+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:45:23.903 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:45:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:45:23.633+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:45:23.903 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:45:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:45:23.633+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:45:23.904 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:45:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:45:23.633+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:45:24.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:45:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:45:23.901+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:45:24.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:45:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:45:23.901+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:45:24.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:45:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:45:23.901+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:45:24.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:45:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:45:23.902+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:45:24.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:45:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:45:24.017+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:45:24.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:45:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:45:24.018+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:45:24.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:45:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:45:24.018+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:45:24.321 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:45:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:45:24.018+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:45:25.032 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:45:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:45:24.679+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:45:25.032 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:45:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:45:24.679+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:45:25.032 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:45:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:45:24.679+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:45:25.032 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:45:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:45:24.680+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:45:25.033 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:45:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:45:24.890+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:45:25.033 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:45:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:45:24.890+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:45:25.033 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:45:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:45:24.890+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:45:25.033 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:45:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:45:24.891+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:45:25.319 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:45:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:45:25.031+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:45:25.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:45:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:45:25.031+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:45:25.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:45:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:45:25.031+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:45:25.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:45:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:45:25.032+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:45:25.819 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:45:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:45:25.502+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 22584 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:45:25.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:45:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:45:25.678+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:45:25.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:45:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:45:25.678+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:45:25.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:45:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:45:25.678+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:45:25.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:45:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:45:25.679+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:45:26.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:45:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:45:25.885+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:45:26.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:45:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:45:25.885+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:45:26.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:45:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:45:25.885+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:45:26.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:45:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:45:25.886+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:45:26.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:45:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:45:26.025+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:45:26.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:45:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:45:26.025+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:45:26.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:45:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:45:26.025+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:45:26.321 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:45:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:45:26.026+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:45:27.013 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:45:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:45:26.640+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:45:27.013 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:45:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:45:26.640+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:45:27.013 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:45:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:45:26.640+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:45:27.013 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:45:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:45:26.641+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:45:27.013 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:45:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:45:26.861+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:45:27.014 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:45:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:45:26.861+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:45:27.014 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:45:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:45:26.861+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:45:27.014 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:45:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:45:26.861+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:45:27.319 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:45:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:45:27.011+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:45:27.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:45:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:45:27.011+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:45:27.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:45:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:45:27.011+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:45:27.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:45:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:45:27.012+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:45:27.973 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:45:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:45:27.866+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:45:27.974 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:45:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:45:27.867+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:45:27.974 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:45:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:45:27.867+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:45:27.974 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:45:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:45:27.867+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:45:27.974 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:45:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:45:27.678+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:45:27.974 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:45:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:45:27.678+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:45:27.975 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:45:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:45:27.678+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:45:27.975 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:45:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:45:27.679+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:45:28.319 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:45:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:45:27.972+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:45:28.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:45:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:45:27.972+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:45:28.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:45:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:45:27.972+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:45:28.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:45:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:45:27.973+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:45:28.943 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:45:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:45:28.829+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:45:28.943 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:45:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:45:28.829+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:45:28.943 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:45:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:45:28.829+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:45:28.943 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:45:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:45:28.830+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:45:28.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:45:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:45:28.684+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:45:28.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:45:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:45:28.684+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:45:28.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:45:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:45:28.684+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:45:28.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:45:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:45:28.685+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:45:29.319 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:45:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:45:28.942+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:45:29.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:45:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:45:28.942+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:45:29.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:45:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:45:28.942+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:45:29.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:45:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:45:28.942+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:45:30.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:45:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:45:29.697+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:45:30.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:45:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:45:29.697+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:45:30.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:45:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:45:29.697+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:45:30.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:45:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:45:29.698+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:45:30.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:45:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:45:29.807+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:45:30.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:45:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:45:29.807+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:45:30.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:45:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:45:29.807+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:45:30.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:45:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:45:29.808+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:45:30.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:45:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:45:29.916+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:45:30.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:45:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:45:29.916+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:45:30.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:45:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:45:29.916+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:45:30.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:45:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:45:29.917+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:45:30.819 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:45:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:45:30.504+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 22667 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:45:30.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:45:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:45:30.725+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:45:30.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:45:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:45:30.725+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:45:30.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:45:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:45:30.725+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:45:30.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:45:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:45:30.726+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:45:31.163 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:45:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:45:30.898+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:45:31.163 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:45:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:45:30.898+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:45:31.164 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:45:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:45:30.898+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:45:31.164 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:45:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:45:30.899+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:45:31.164 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:45:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:45:30.846+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:45:31.164 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:45:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:45:30.846+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:45:31.164 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:45:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:45:30.846+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:45:31.164 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:45:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:45:30.846+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:45:32.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:45:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:45:31.698+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:45:32.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:45:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:45:31.698+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:45:32.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:45:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:45:31.698+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:45:32.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:45:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:45:31.699+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:45:32.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:45:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:45:31.879+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:45:32.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:45:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:45:31.879+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:45:32.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:45:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:45:31.879+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:45:32.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:45:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:45:31.880+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:45:32.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:45:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:45:31.878+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:45:32.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:45:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:45:31.879+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:45:32.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:45:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:45:31.879+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:45:32.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:45:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:45:31.879+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:45:33.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:45:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:45:32.686+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:45:33.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:45:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:45:32.686+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:45:33.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:45:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:45:32.686+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:45:33.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:45:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:45:32.687+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:45:33.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:45:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:45:32.832+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:45:33.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:45:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:45:32.832+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:45:33.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:45:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:45:32.832+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:45:33.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:45:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:45:32.833+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:45:33.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:45:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:45:32.856+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:45:33.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:45:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:45:32.856+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:45:33.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:45:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:45:32.856+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:45:33.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:45:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:45:32.856+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:45:34.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:45:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:45:33.732+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:45:34.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:45:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:45:33.732+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:45:34.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:45:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:45:33.732+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:45:34.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:45:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:45:33.733+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:45:34.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:45:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:45:33.784+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:45:34.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:45:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:45:33.784+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:45:34.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:45:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:45:33.784+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:45:34.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:45:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:45:33.784+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:45:34.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:45:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:45:33.834+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:45:34.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:45:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:45:33.834+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:45:34.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:45:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:45:33.834+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:45:34.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:45:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:45:33.835+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:45:35.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:45:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:45:34.705+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:45:35.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:45:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:45:34.705+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:45:35.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:45:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:45:34.705+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:45:35.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:45:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:45:34.705+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:45:35.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:45:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:45:34.751+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:45:35.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:45:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:45:34.752+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:45:35.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:45:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:45:34.752+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:45:35.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:45:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:45:34.752+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:45:35.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:45:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:45:34.819+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:45:35.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:45:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:45:34.819+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:45:35.072 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:45:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:45:34.819+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:45:35.072 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:45:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:45:34.819+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:45:35.801 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:45:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:45:35.506+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 22750 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:45:35.801 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:45:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:45:35.715+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:45:35.802 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:45:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:45:35.715+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:45:35.802 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:45:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:45:35.715+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:45:35.802 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:45:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:45:35.715+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:45:35.802 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:45:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:45:35.679+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:45:35.802 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:45:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:45:35.679+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:45:35.802 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:45:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:45:35.679+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:45:35.803 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:45:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:45:35.680+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:45:36.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:45:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:45:35.799+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:45:36.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:45:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:45:35.799+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:45:36.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:45:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:45:35.799+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:45:36.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:45:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:45:35.799+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:45:37.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:45:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:45:36.678+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:45:37.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:45:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:45:36.678+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:45:37.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:45:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:45:36.678+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:45:37.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:45:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:45:36.679+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:45:37.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:45:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:45:36.716+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:45:37.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:45:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:45:36.716+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:45:37.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:45:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:45:36.716+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:45:37.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:45:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:45:36.717+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:45:37.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:45:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:45:36.831+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:45:37.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:45:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:45:36.831+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:45:37.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:45:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:45:36.831+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:45:37.072 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:45:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:45:36.832+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:45:38.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:45:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:45:37.637+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:45:38.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:45:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:45:37.637+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:45:38.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:45:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:45:37.637+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:45:38.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:45:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:45:37.638+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:45:38.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:45:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:45:37.675+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:45:38.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:45:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:45:37.675+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:45:38.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:45:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:45:37.675+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:45:38.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:45:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:45:37.676+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:45:38.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:45:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:45:37.799+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:45:38.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:45:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:45:37.799+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:45:38.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:45:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:45:37.799+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:45:38.072 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:45:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:45:37.800+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:45:39.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:45:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:45:38.654+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:45:39.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:45:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:45:38.654+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:45:39.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:45:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:45:38.654+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:45:39.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:45:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:45:38.655+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:45:39.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:45:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:45:38.630+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:45:39.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:45:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:45:38.630+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:45:39.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:45:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:45:38.630+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:45:39.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:45:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:45:38.630+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:45:39.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:45:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:45:38.785+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:45:39.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:45:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:45:38.785+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:45:39.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:45:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:45:38.785+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:45:39.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:45:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:45:38.785+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:45:40.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:45:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:45:39.623+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:45:40.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:45:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:45:39.623+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:45:40.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:45:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:45:39.623+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:45:40.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:45:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:45:39.624+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:45:40.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:45:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:45:39.586+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:45:40.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:45:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:45:39.586+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:45:40.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:45:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:45:39.586+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:45:40.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:45:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:45:39.586+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:45:40.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:45:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:45:39.748+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:45:40.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:45:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:45:39.748+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:45:40.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:45:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:45:39.748+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:45:40.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:45:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:45:39.749+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:45:40.762 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:45:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:45:40.508+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 22833 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:45:40.763 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:45:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:45:40.597+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:45:40.763 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:45:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:45:40.597+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:45:40.763 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:45:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:45:40.597+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:45:40.763 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:45:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:45:40.598+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:45:40.764 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:45:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:45:40.563+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:45:40.764 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:45:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:45:40.563+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:45:40.764 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:45:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:45:40.563+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:45:40.764 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:45:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:45:40.563+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:45:41.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:45:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:45:40.760+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:45:41.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:45:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:45:40.760+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:45:41.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:45:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:45:40.760+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:45:41.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:45:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:45:40.761+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:45:41.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:45:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:45:41.583+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:45:41.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:45:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:45:41.583+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:45:41.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:45:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:45:41.583+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:45:41.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:45:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:45:41.583+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:45:41.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:45:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:45:41.734+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:45:41.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:45:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:45:41.734+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:45:41.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:45:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:45:41.734+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:45:41.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:45:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:45:41.735+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:45:41.821 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:45:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:45:41.520+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:45:41.821 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:45:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:45:41.520+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:45:41.821 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:45:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:45:41.520+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:45:41.822 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:45:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:45:41.521+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:45:42.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:45:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:45:42.536+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:45:42.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:45:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:45:42.536+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:45:42.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:45:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:45:42.536+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:45:42.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:45:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:45:42.536+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:45:42.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:45:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:45:42.767+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:45:42.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:45:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:45:42.767+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:45:42.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:45:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:45:42.767+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:45:42.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:45:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:45:42.767+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:45:42.821 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:45:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:45:42.544+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:45:42.821 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:45:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:45:42.544+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:45:42.821 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:45:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:45:42.544+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:45:42.822 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:45:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:45:42.545+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:45:43.777 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:45:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:45:43.567+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:45:43.778 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:45:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:45:43.567+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:45:43.778 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:45:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:45:43.567+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:45:43.778 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:45:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:45:43.568+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:45:43.778 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:45:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:45:43.569+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:45:43.778 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:45:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:45:43.569+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:45:43.778 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:45:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:45:43.569+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:45:43.778 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:45:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:45:43.569+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:45:44.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:45:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:45:43.776+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:45:44.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:45:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:45:43.776+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:45:44.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:45:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:45:43.776+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:45:44.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:45:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:45:43.776+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:45:44.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:45:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:45:44.728+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:45:44.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:45:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:45:44.728+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:45:44.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:45:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:45:44.728+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:45:44.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:45:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:45:44.728+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:45:44.821 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:45:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:45:44.580+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:45:44.821 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:45:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:45:44.580+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:45:44.821 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:45:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:45:44.580+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:45:44.821 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:45:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:45:44.581+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:45:44.821 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:45:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:45:44.539+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:45:44.821 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:45:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:45:44.539+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:45:44.821 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:45:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:45:44.539+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:45:44.822 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:45:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:45:44.540+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:45:45.820 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:45:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:45:45.511+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 22916 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:45:45.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:45:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:45:45.554+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:45:45.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:45:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:45:45.554+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:45:45.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:45:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:45:45.554+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:45:45.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:45:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:45:45.555+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:45:45.821 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:45:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:45:45.534+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:45:45.821 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:45:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:45:45.534+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:45:45.821 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:45:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:45:45.534+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:45:45.821 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:45:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:45:45.535+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:45:45.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:45:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:45:45.692+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:45:45.822 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:45:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:45:45.692+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:45:45.822 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:45:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:45:45.692+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:45:45.822 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:45:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:45:45.692+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:45:46.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:45:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:45:46.728+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:45:46.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:45:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:45:46.728+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:45:46.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:45:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:45:46.728+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:45:46.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:45:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:45:46.729+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:45:46.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:45:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:45:46.522+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:45:46.821 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:45:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:45:46.522+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:45:46.821 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:45:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:45:46.522+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:45:46.821 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:45:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:45:46.523+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:45:46.821 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:45:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:45:46.517+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:45:46.821 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:45:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:45:46.517+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:45:46.821 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:45:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:45:46.517+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:45:46.821 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:45:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:45:46.518+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:45:47.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:45:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:45:47.538+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:45:47.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:45:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:45:47.538+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:45:47.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:45:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:45:47.538+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:45:47.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:45:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:45:47.539+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:45:47.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:45:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:45:47.730+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:45:47.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:45:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:45:47.730+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:45:47.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:45:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:45:47.730+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:45:47.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:45:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:45:47.731+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:45:47.821 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:45:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:45:47.505+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:45:47.822 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:45:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:45:47.505+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:45:47.822 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:45:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:45:47.505+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:45:47.822 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:45:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:45:47.506+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:45:48.779 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:45:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:45:48.472+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:45:48.779 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:45:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:45:48.472+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:45:48.779 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:45:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:45:48.472+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:45:48.780 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:45:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:45:48.472+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:45:48.780 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:45:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:45:48.501+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:45:48.780 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:45:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:45:48.501+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:45:48.780 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:45:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:45:48.501+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:45:48.780 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:45:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:45:48.501+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:45:49.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:45:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:45:48.778+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:45:49.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:45:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:45:48.778+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:45:49.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:45:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:45:48.778+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:45:49.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:45:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:45:48.778+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:45:49.790 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:45:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:45:49.477+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:45:49.790 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:45:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:45:49.477+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:45:49.790 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:45:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:45:49.477+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:45:49.791 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:45:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:45:49.478+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:45:49.791 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:45:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:45:49.492+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:45:49.791 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:45:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:45:49.492+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:45:49.791 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:45:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:45:49.492+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:45:49.791 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:45:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:45:49.492+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:45:50.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:45:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:45:49.789+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:45:50.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:45:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:45:49.789+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:45:50.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:45:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:45:49.789+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:45:50.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:45:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:45:49.789+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:45:50.819 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:45:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:45:50.514+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 23012 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:45:50.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:45:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:45:50.444+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:45:50.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:45:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:45:50.444+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:45:50.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:45:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:45:50.444+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:45:50.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:45:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:45:50.446+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:45:50.821 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:45:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:45:50.448+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:45:50.821 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:45:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:45:50.448+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:45:50.821 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:45:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:45:50.448+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:45:50.821 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:45:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:45:50.449+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:45:51.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:45:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:45:50.830+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:45:51.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:45:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:45:50.830+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:45:51.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:45:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:45:50.830+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:45:51.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:45:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:45:50.831+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:45:51.721 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:45:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:45:51.404+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:45:51.721 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:45:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:45:51.404+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:45:51.721 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:45:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:45:51.404+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:45:51.721 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:45:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:45:51.404+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:45:51.722 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:45:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:45:51.426+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:45:51.722 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:45:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:45:51.426+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:45:51.722 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:45:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:45:51.426+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:45:51.722 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:45:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:45:51.426+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:45:52.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:45:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:45:51.819+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:45:52.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:45:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:45:51.819+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:45:52.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:45:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:45:51.819+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:45:52.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:45:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:45:51.819+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:45:52.667 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:45:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:45:52.426+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:45:52.667 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:45:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:45:52.426+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:45:52.667 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:45:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:45:52.426+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:45:52.667 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:45:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:45:52.426+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:45:52.668 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:45:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:45:52.389+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:45:52.668 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:45:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:45:52.389+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:45:52.668 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:45:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:45:52.389+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:45:52.668 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:45:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:45:52.390+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:45:53.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:45:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:45:52.860+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:45:53.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:45:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:45:52.860+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:45:53.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:45:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:45:52.860+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:45:53.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:45:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:45:52.861+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:45:53.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:45:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:45:53.396+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:45:53.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:45:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:45:53.396+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:45:53.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:45:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:45:53.396+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:45:53.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:45:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:45:53.396+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:45:53.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:45:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:45:53.375+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:45:53.821 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:45:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:45:53.375+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:45:53.821 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:45:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:45:53.375+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:45:53.821 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:45:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:45:53.376+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:45:54.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:45:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:45:53.846+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:45:54.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:45:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:45:53.846+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:45:54.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:45:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:45:53.846+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:45:54.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:45:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:45:53.847+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:45:54.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:45:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:45:54.332+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:45:54.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:45:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:45:54.332+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:45:54.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:45:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:45:54.332+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:45:54.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:45:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:45:54.333+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:45:54.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:45:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:45:54.407+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:45:54.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:45:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:45:54.407+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:45:54.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:45:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:45:54.407+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:45:54.821 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:45:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:45:54.408+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:45:55.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:45:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:45:54.847+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:45:55.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:45:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:45:54.847+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:45:55.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:45:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:45:54.847+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:45:55.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:45:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:45:54.847+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:45:55.638 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:45:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:45:55.516+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 23095 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:45:55.639 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:45:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:45:55.412+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:45:55.639 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:45:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:45:55.412+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:45:55.639 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:45:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:45:55.412+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:45:55.639 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:45:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:45:55.412+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:45:55.639 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:45:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:45:55.343+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:45:55.640 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:45:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:45:55.343+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:45:55.640 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:45:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:45:55.343+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:45:55.640 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:45:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:45:55.343+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:45:56.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:45:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:45:55.813+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:45:56.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:45:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:45:55.813+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:45:56.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:45:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:45:55.813+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:45:56.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:45:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:45:55.814+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:45:56.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:45:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:45:56.402+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:45:56.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:45:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:45:56.402+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:45:56.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:45:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:45:56.402+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:45:56.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:45:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:45:56.403+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:45:56.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:45:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:45:56.347+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:45:56.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:45:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:45:56.347+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:45:56.821 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:45:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:45:56.347+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:45:56.821 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:45:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:45:56.348+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:45:57.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:45:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:45:56.858+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:45:57.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:45:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:45:56.858+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:45:57.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:45:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:45:56.858+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:45:57.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:45:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:45:56.858+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:45:57.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:45:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:45:57.403+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:45:57.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:45:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:45:57.403+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:45:57.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:45:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:45:57.403+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:45:57.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:45:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:45:57.404+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:45:57.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:45:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:45:57.394+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:45:57.821 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:45:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:45:57.394+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:45:57.821 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:45:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:45:57.394+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:45:57.821 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:45:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:45:57.395+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:45:58.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:45:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:45:57.907+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:45:58.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:45:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:45:57.907+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:45:58.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:45:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:45:57.907+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:45:58.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:45:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:45:57.907+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:45:58.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:45:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:45:58.425+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:45:58.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:45:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:45:58.425+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:45:58.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:45:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:45:58.425+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:45:58.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:45:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:45:58.426+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:45:58.821 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:45:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:45:58.382+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:45:58.821 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:45:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:45:58.382+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:45:58.821 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:45:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:45:58.382+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:45:58.821 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:45:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:45:58.382+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:45:59.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:45:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:45:58.891+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:45:59.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:45:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:45:58.891+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:45:59.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:45:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:45:58.891+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:45:59.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:45:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:45:58.892+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:45:59.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:45:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:45:59.474+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:45:59.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:45:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:45:59.474+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:45:59.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:45:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:45:59.474+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:45:59.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:45:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:45:59.475+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:45:59.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:45:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:45:59.364+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:45:59.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:45:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:45:59.364+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:45:59.821 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:45:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:45:59.364+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:45:59.821 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:45:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:45:59.365+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:46:00.190 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:45:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:45:59.855+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:46:00.190 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:45:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:45:59.855+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:46:00.190 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:45:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:45:59.855+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:46:00.191 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:45:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:45:59.855+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:46:00.453 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:46:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:46:00.329+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:46:00.454 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:46:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:46:00.329+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:46:00.454 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:46:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:46:00.329+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:46:00.454 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:46:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:46:00.330+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:46:00.820 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:46:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:46:00.519+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 23178 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:46:00.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:46:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:46:00.452+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:46:00.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:46:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:46:00.452+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:46:00.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:46:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:46:00.452+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:46:00.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:46:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:46:00.453+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:46:01.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:46:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:46:00.875+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:46:01.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:46:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:46:00.875+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:46:01.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:46:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:46:00.875+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:46:01.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:46:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:46:00.876+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:46:01.722 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:46:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:46:01.334+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:46:01.722 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:46:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:46:01.334+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:46:01.722 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:46:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:46:01.334+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:46:01.722 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:46:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:46:01.335+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:46:01.722 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:46:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:46:01.497+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:46:01.723 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:46:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:46:01.497+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:46:01.723 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:46:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:46:01.497+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:46:01.723 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:46:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:46:01.497+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:46:02.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:46:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:46:01.880+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:46:02.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:46:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:46:01.880+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:46:02.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:46:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:46:01.880+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:46:02.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:46:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:46:01.880+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:46:02.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:46:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:46:02.368+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:46:02.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:46:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:46:02.368+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:46:02.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:46:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:46:02.368+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:46:02.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:46:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:46:02.369+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:46:02.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:46:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:46:02.535+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:46:02.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:46:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:46:02.535+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:46:02.821 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:46:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:46:02.535+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:46:02.821 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:46:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:46:02.536+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:46:03.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:46:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:46:02.860+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:46:03.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:46:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:46:02.860+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:46:03.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:46:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:46:02.860+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:46:03.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:46:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:46:02.861+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:46:03.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:46:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:46:03.491+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:46:03.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:46:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:46:03.491+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:46:03.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:46:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:46:03.491+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:46:03.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:46:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:46:03.492+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:46:03.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:46:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:46:03.382+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:46:03.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:46:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:46:03.382+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:46:03.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:46:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:46:03.382+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:46:03.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:46:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:46:03.383+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:46:04.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:46:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:46:03.854+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:46:04.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:46:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:46:03.854+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:46:04.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:46:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:46:03.854+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:46:04.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:46:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:46:03.854+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:46:04.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:46:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:46:04.428+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:46:04.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:46:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:46:04.428+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:46:04.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:46:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:46:04.428+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:46:04.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:46:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:46:04.428+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:46:04.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:46:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:46:04.511+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:46:04.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:46:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:46:04.511+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:46:04.821 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:46:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:46:04.511+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:46:04.821 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:46:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:46:04.512+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:46:05.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:46:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:46:04.819+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:46:05.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:46:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:46:04.819+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:46:05.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:46:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:46:04.819+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:46:05.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:46:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:46:04.820+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:46:05.815 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:46:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:46:05.521+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 23260 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:46:05.816 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:46:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:46:05.512+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:46:05.816 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:46:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:46:05.512+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:46:05.816 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:46:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:46:05.512+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:46:05.816 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:46:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:46:05.513+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:46:05.816 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:46:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:46:05.402+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:46:05.816 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:46:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:46:05.402+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:46:05.817 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:46:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:46:05.402+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:46:05.817 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:46:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:46:05.403+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:46:06.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:46:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:46:05.814+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:46:06.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:46:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:46:05.814+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:46:06.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:46:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:46:05.814+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:46:06.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:46:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:46:05.815+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:46:06.817 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:46:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:46:06.488+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:46:06.818 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:46:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:46:06.488+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:46:06.818 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:46:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:46:06.488+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:46:06.818 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:46:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:46:06.489+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:46:06.818 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:46:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:46:06.427+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:46:06.818 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:46:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:46:06.427+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:46:06.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:46:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:46:06.427+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:46:06.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:46:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:46:06.428+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:46:07.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:46:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:46:06.816+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:46:07.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:46:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:46:06.816+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:46:07.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:46:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:46:06.816+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:46:07.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:46:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:46:06.816+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:46:07.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:46:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:46:07.479+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:46:07.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:46:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:46:07.479+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:46:07.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:46:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:46:07.479+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:46:07.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:46:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:46:07.480+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:46:07.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:46:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:46:07.395+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:46:07.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:46:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:46:07.395+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:46:07.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:46:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:46:07.395+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:46:07.821 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:46:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:46:07.396+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:46:08.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:46:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:46:07.836+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:46:08.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:46:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:46:07.836+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:46:08.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:46:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:46:07.836+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:46:08.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:46:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:46:07.837+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:46:08.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:46:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:46:08.347+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:46:08.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:46:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:46:08.347+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:46:08.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:46:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:46:08.347+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:46:08.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:46:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:46:08.348+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:46:08.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:46:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:46:08.461+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:46:08.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:46:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:46:08.461+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:46:08.821 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:46:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:46:08.461+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:46:08.821 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:46:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:46:08.462+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:46:09.116 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:46:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:46:08.839+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:46:09.117 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:46:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:46:08.839+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:46:09.117 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:46:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:46:08.839+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:46:09.117 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:46:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:46:08.840+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:46:09.491 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:46:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:46:09.341+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:46:09.491 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:46:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:46:09.341+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:46:09.492 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:46:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:46:09.341+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:46:09.492 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:46:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:46:09.341+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:46:09.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:46:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:46:09.490+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:46:09.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:46:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:46:09.490+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:46:09.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:46:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:46:09.490+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:46:09.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:46:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:46:09.491+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:46:10.191 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:46:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:46:09.880+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:46:10.191 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:46:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:46:09.880+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:46:10.191 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:46:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:46:09.880+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:46:10.192 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:46:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:46:09.880+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:46:10.496 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:46:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:46:10.326+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:46:10.496 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:46:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:46:10.326+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:46:10.496 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:46:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:46:10.326+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:46:10.496 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:46:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:46:10.327+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:46:10.819 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:46:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:46:10.523+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 23343 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:46:10.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:46:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:46:10.495+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:46:10.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:46:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:46:10.495+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:46:10.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:46:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:46:10.495+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:46:10.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:46:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:46:10.495+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:46:11.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:46:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:46:10.883+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:46:11.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:46:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:46:10.883+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:46:11.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:46:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:46:10.883+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:46:11.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:46:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:46:10.883+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:46:11.722 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:46:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:46:11.335+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:46:11.722 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:46:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:46:11.335+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:46:11.722 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:46:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:46:11.335+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:46:11.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:46:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:46:11.336+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:46:11.723 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:46:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:46:11.533+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:46:11.723 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:46:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:46:11.533+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:46:11.723 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:46:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:46:11.533+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:46:11.723 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:46:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:46:11.534+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:46:12.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:46:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:46:11.836+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:46:12.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:46:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:46:11.836+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:46:12.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:46:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:46:11.836+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:46:12.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:46:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:46:11.836+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:46:12.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:46:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:46:12.323+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:46:12.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:46:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:46:12.323+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:46:12.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:46:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:46:12.323+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:46:12.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:46:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:46:12.324+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:46:12.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:46:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:46:12.504+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:46:12.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:46:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:46:12.504+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:46:12.821 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:46:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:46:12.504+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:46:12.821 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:46:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:46:12.505+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:46:13.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:46:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:46:12.827+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:46:13.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:46:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:46:12.827+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:46:13.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:46:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:46:12.827+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:46:13.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:46:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:46:12.828+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:46:13.797 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:46:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:46:13.539+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:46:13.798 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:46:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:46:13.539+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:46:13.798 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:46:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:46:13.539+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:46:13.798 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:46:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:46:13.540+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:46:13.798 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:46:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:46:13.336+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:46:13.798 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:46:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:46:13.336+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:46:13.799 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:46:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:46:13.336+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:46:13.799 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:46:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:46:13.336+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:46:14.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:46:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:46:13.796+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:46:14.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:46:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:46:13.796+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:46:14.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:46:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:46:13.796+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:46:14.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:46:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:46:13.797+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:46:14.491 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:46:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:46:14.376+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:46:14.492 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:46:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:46:14.376+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:46:14.492 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:46:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:46:14.376+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:46:14.492 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:46:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:46:14.377+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:46:14.807 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:46:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:46:14.490+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:46:14.807 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:46:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:46:14.490+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:46:14.807 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:46:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:46:14.490+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:46:14.807 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:46:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:46:14.491+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:46:15.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:46:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:46:14.805+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:46:15.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:46:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:46:14.805+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:46:15.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:46:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:46:14.806+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:46:15.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:46:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:46:14.806+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:46:15.650 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:46:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:46:15.526+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 23426 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:46:15.651 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:46:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:46:15.357+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:46:15.651 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:46:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:46:15.357+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:46:15.651 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:46:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:46:15.357+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:46:15.651 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:46:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:46:15.357+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:46:15.651 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:46:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:46:15.530+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:46:15.651 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:46:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:46:15.530+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:46:15.652 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:46:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:46:15.530+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:46:15.652 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:46:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:46:15.531+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:46:16.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:46:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:46:15.779+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:46:16.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:46:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:46:15.779+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:46:16.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:46:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:46:15.779+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:46:16.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:46:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:46:15.779+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:46:16.780 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:46:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:46:16.536+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:46:16.781 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:46:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:46:16.536+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:46:16.781 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:46:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:46:16.536+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:46:16.781 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:46:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:46:16.536+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:46:16.781 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:46:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:46:16.385+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:46:16.781 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:46:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:46:16.385+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:46:16.781 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:46:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:46:16.385+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:46:16.781 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:46:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:46:16.385+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:46:17.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:46:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:46:16.779+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:46:17.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:46:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:46:16.779+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:46:17.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:46:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:46:16.779+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:46:17.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:46:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:46:16.780+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:46:17.757 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:46:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:46:17.498+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:46:17.758 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:46:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:46:17.498+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:46:17.758 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:46:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:46:17.498+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:46:17.758 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:46:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:46:17.499+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:46:17.758 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:46:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:46:17.379+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:46:17.758 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:46:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:46:17.379+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:46:17.759 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:46:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:46:17.379+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:46:17.759 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:46:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:46:17.380+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:46:18.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:46:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:46:17.756+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:46:18.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:46:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:46:17.756+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:46:18.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:46:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:46:17.756+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:46:18.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:46:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:46:17.757+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:46:18.781 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:46:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:46:18.392+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:46:18.782 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:46:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:46:18.392+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:46:18.782 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:46:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:46:18.392+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:46:18.782 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:46:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:46:18.392+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:46:18.783 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:46:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:46:18.474+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:46:18.783 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:46:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:46:18.474+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:46:18.783 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:46:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:46:18.474+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:46:18.784 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:46:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:46:18.475+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:46:19.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:46:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:46:18.780+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:46:19.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:46:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:46:18.780+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:46:19.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:46:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:46:18.780+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:46:19.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:46:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:46:18.780+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:46:19.773 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:46:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:46:19.426+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:46:19.773 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:46:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:46:19.426+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:46:19.774 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:46:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:46:19.426+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:46:19.774 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:46:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:46:19.426+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:46:19.774 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:46:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:46:19.436+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:46:19.774 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:46:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:46:19.436+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:46:19.774 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:46:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:46:19.436+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:46:19.774 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:46:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:46:19.437+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:46:20.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:46:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:46:19.772+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:46:20.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:46:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:46:19.772+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:46:20.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:46:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:46:19.772+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:46:20.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:46:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:46:19.772+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:46:20.484 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:46:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:46:20.417+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:46:20.485 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:46:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:46:20.417+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:46:20.485 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:46:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:46:20.417+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:46:20.485 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:46:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:46:20.418+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:46:20.801 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:46:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:46:20.529+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 23509 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:46:20.802 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:46:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:46:20.483+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:46:20.802 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:46:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:46:20.483+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:46:20.802 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:46:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:46:20.483+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:46:20.802 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:46:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:46:20.484+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:46:21.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:46:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:46:20.800+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:46:21.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:46:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:46:20.800+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:46:21.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:46:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:46:20.800+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:46:21.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:46:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:46:20.801+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:46:21.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:46:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:46:21.525+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:46:21.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:46:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:46:21.525+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:46:21.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:46:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:46:21.525+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:46:21.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:46:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:46:21.525+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:46:21.723 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:46:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:46:21.408+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:46:21.724 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:46:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:46:21.408+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:46:21.724 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:46:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:46:21.408+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:46:21.724 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:46:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:46:21.409+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:46:22.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:46:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:46:21.788+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:46:22.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:46:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:46:21.788+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:46:22.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:46:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:46:21.788+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:46:22.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:46:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:46:21.789+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:46:22.796 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:46:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:46:22.564+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:46:22.796 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:46:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:46:22.564+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:46:22.796 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:46:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:46:22.564+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:46:22.796 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:46:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:46:22.565+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:46:22.797 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:46:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:46:22.419+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:46:22.797 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:46:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:46:22.419+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:46:22.797 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:46:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:46:22.419+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:46:22.797 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:46:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:46:22.420+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:46:23.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:46:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:46:22.794+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:46:23.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:46:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:46:22.794+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:46:23.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:46:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:46:22.794+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:46:23.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:46:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:46:22.795+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:46:23.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:46:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:46:23.470+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:46:23.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:46:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:46:23.470+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:46:23.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:46:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:46:23.470+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:46:23.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:46:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:46:23.471+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:46:23.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:46:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:46:23.524+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:46:23.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:46:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:46:23.524+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:46:23.821 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:46:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:46:23.524+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:46:23.821 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:46:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:46:23.525+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:46:24.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:46:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:46:23.828+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:46:24.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:46:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:46:23.828+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:46:24.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:46:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:46:23.828+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:46:24.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:46:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:46:23.829+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:46:24.783 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:46:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:46:24.507+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:46:24.783 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:46:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:46:24.507+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:46:24.783 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:46:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:46:24.507+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:46:24.783 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:46:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:46:24.508+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:46:24.783 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:46:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:46:24.488+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:46:24.784 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:46:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:46:24.488+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:46:24.784 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:46:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:46:24.488+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:46:24.784 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:46:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:46:24.488+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:46:25.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:46:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:46:24.781+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:46:25.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:46:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:46:24.781+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:46:25.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:46:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:46:24.781+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:46:25.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:46:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:46:24.782+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:46:25.820 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:46:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:46:25.531+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 23592 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:46:25.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:46:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:46:25.498+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:46:25.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:46:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:46:25.498+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:46:25.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:46:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:46:25.498+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:46:25.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:46:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:46:25.498+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:46:25.821 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:46:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:46:25.462+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:46:25.821 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:46:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:46:25.462+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:46:25.821 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:46:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:46:25.462+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:46:25.821 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:46:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:46:25.462+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:46:26.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:46:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:46:25.827+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:46:26.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:46:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:46:25.827+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:46:26.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:46:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:46:25.827+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:46:26.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:46:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:46:25.828+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:46:26.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:46:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:46:26.444+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:46:26.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:46:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:46:26.444+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:46:26.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:46:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:46:26.444+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:46:26.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:46:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:46:26.445+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:46:26.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:46:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:46:26.511+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:46:26.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:46:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:46:26.511+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:46:26.821 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:46:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:46:26.511+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:46:26.821 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:46:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:46:26.511+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:46:27.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:46:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:46:26.844+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:46:27.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:46:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:46:26.844+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:46:27.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:46:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:46:26.844+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:46:27.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:46:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:46:26.845+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:46:27.813 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:46:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:46:27.433+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:46:27.813 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:46:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:46:27.433+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:46:27.814 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:46:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:46:27.433+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:46:27.814 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:46:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:46:27.434+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:46:27.814 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:46:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:46:27.531+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:46:27.814 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:46:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:46:27.531+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:46:27.814 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:46:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:46:27.531+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:46:27.815 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:46:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:46:27.532+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:46:28.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:46:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:46:27.812+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:46:28.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:46:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:46:27.812+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:46:28.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:46:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:46:27.812+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:46:28.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:46:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:46:27.812+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:46:28.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:46:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:46:28.482+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:46:28.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:46:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:46:28.482+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:46:28.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:46:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:46:28.482+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:46:28.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:46:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:46:28.483+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:46:28.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:46:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:46:28.493+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:46:28.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:46:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:46:28.493+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:46:28.821 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:46:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:46:28.493+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:46:28.821 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:46:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:46:28.493+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:46:29.116 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:46:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:46:28.844+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:46:29.116 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:46:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:46:28.844+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:46:29.117 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:46:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:46:28.844+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:46:29.117 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:46:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:46:28.844+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:46:29.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:46:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:46:29.448+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:46:29.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:46:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:46:29.448+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:46:29.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:46:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:46:29.448+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:46:29.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:46:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:46:29.449+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:46:29.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:46:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:46:29.515+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:46:29.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:46:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:46:29.515+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:46:29.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:46:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:46:29.515+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:46:29.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:46:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:46:29.516+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:46:30.193 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:46:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:46:29.884+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:46:30.193 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:46:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:46:29.884+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:46:30.193 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:46:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:46:29.884+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:46:30.193 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:46:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:46:29.885+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:46:30.819 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:46:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:46:30.533+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 23675 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:46:30.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:46:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:46:30.563+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:46:30.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:46:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:46:30.563+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:46:30.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:46:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:46:30.563+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:46:30.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:46:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:46:30.563+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:46:30.821 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:46:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:46:30.478+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:46:30.821 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:46:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:46:30.478+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:46:30.821 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:46:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:46:30.478+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:46:30.821 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:46:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:46:30.479+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:46:31.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:46:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:46:30.880+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:46:31.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:46:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:46:30.880+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:46:31.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:46:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:46:30.880+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:46:31.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:46:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:46:30.880+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:46:31.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:46:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:46:31.473+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:46:31.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:46:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:46:31.473+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:46:31.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:46:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:46:31.473+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:46:31.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:46:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:46:31.473+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:46:31.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:46:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:46:31.597+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:46:31.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:46:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:46:31.597+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:46:31.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:46:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:46:31.597+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:46:31.821 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:46:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:46:31.598+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:46:32.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:46:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:46:31.839+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:46:32.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:46:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:46:31.839+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:46:32.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:46:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:46:31.839+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:46:32.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:46:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:46:31.839+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:46:32.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:46:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:46:32.474+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:46:32.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:46:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:46:32.474+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:46:32.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:46:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:46:32.474+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:46:32.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:46:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:46:32.475+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:46:32.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:46:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:46:32.624+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:46:32.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:46:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:46:32.624+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:46:32.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:46:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:46:32.624+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:46:32.821 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:46:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:46:32.625+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:46:33.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:46:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:46:32.828+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:46:33.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:46:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:46:32.828+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:46:33.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:46:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:46:32.828+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:46:33.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:46:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:46:32.828+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:46:33.797 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:46:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:46:33.440+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:46:33.797 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:46:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:46:33.440+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:46:33.797 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:46:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:46:33.440+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:46:33.797 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:46:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:46:33.440+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:46:33.797 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:46:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:46:33.586+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:46:33.798 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:46:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:46:33.586+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:46:33.798 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:46:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:46:33.586+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:46:33.798 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:46:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:46:33.586+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:46:34.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:46:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:46:33.794+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:46:34.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:46:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:46:33.794+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:46:34.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:46:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:46:33.794+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:46:34.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:46:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:46:33.795+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:46:34.767 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:46:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:46:34.397+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:46:34.767 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:46:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:46:34.397+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:46:34.768 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:46:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:46:34.397+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:46:34.768 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:46:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:46:34.397+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:46:34.768 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:46:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:46:34.608+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:46:34.768 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:46:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:46:34.608+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:46:34.768 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:46:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:46:34.608+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:46:34.768 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:46:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:46:34.608+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:46:35.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:46:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:46:34.765+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:46:35.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:46:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:46:34.765+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:46:35.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:46:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:46:34.765+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:46:35.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:46:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:46:34.765+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:46:35.719 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:46:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:46:35.535+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 23758 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:46:35.719 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:46:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:46:35.423+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:46:35.719 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:46:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:46:35.423+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:46:35.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:46:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:46:35.423+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:46:35.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:46:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:46:35.424+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:46:35.720 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:46:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:46:35.649+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:46:35.720 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:46:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:46:35.649+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:46:35.720 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:46:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:46:35.649+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:46:35.720 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:46:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:46:35.650+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:46:36.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:46:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:46:35.717+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:46:36.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:46:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:46:35.717+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:46:36.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:46:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:46:35.717+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:46:36.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:46:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:46:35.717+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:46:36.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:46:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:46:36.462+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:46:36.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:46:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:46:36.462+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:46:36.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:46:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:46:36.462+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:46:36.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:46:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:46:36.463+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:46:36.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:46:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:46:36.652+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:46:36.821 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:46:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:46:36.652+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:46:36.821 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:46:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:46:36.652+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:46:36.821 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:46:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:46:36.653+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:46:36.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:46:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:46:36.690+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:46:36.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:46:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:46:36.690+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:46:36.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:46:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:46:36.690+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:46:36.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:46:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:46:36.690+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:46:37.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:46:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:46:37.474+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:46:37.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:46:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:46:37.474+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:46:37.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:46:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:46:37.474+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:46:37.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:46:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:46:37.474+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:46:37.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:46:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:46:37.716+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:46:37.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:46:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:46:37.716+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:46:37.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:46:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:46:37.716+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:46:37.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:46:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:46:37.717+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:46:37.821 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:46:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:46:37.647+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:46:37.821 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:46:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:46:37.647+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:46:37.821 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:46:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:46:37.647+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:46:37.822 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:46:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:46:37.647+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:46:38.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:46:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:46:38.523+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:46:38.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:46:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:46:38.523+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:46:38.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:46:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:46:38.523+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:46:38.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:46:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:46:38.524+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:46:38.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:46:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:46:38.629+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:46:38.821 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:46:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:46:38.629+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:46:38.821 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:46:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:46:38.629+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:46:38.821 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:46:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:46:38.629+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:46:38.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:46:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:46:38.727+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:46:38.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:46:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:46:38.727+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:46:38.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:46:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:46:38.727+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:46:38.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:46:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:46:38.727+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:46:39.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:46:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:46:39.671+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:46:39.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:46:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:46:39.671+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:46:39.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:46:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:46:39.671+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:46:39.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:46:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:46:39.672+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:46:39.821 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:46:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:46:39.541+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:46:39.821 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:46:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:46:39.541+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:46:39.821 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:46:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:46:39.541+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:46:39.821 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:46:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:46:39.541+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:46:39.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:46:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:46:39.688+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:46:39.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:46:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:46:39.688+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:46:39.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:46:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:46:39.688+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:46:39.822 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:46:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:46:39.688+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:46:40.819 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:46:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:46:40.537+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 23841 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:46:40.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:46:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:46:40.629+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:46:40.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:46:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:46:40.629+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:46:40.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:46:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:46:40.629+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:46:40.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:46:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:46:40.630+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:46:40.821 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:46:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:46:40.572+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:46:40.821 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:46:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:46:40.572+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:46:40.821 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:46:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:46:40.572+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:46:40.821 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:46:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:46:40.573+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:46:40.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:46:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:46:40.703+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:46:40.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:46:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:46:40.703+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:46:40.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:46:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:46:40.703+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:46:40.822 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:46:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:46:40.703+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:46:41.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:46:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:46:41.624+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:46:41.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:46:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:46:41.624+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:46:41.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:46:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:46:41.624+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:46:41.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:46:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:46:41.624+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:46:41.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:46:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:46:41.535+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:46:41.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:46:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:46:41.535+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:46:41.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:46:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:46:41.535+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:46:41.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:46:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:46:41.535+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:46:41.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:46:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:46:41.670+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:46:41.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:46:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:46:41.670+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:46:41.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:46:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:46:41.670+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:46:41.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:46:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:46:41.670+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:46:42.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:46:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:46:42.575+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:46:42.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:46:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:46:42.575+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:46:42.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:46:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:46:42.575+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:46:42.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:46:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:46:42.576+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:46:42.821 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:46:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:46:42.504+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:46:42.821 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:46:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:46:42.504+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:46:42.821 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:46:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:46:42.504+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:46:42.821 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:46:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:46:42.505+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:46:42.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:46:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:46:42.717+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:46:42.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:46:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:46:42.717+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:46:42.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:46:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:46:42.717+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:46:42.822 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:46:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:46:42.717+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:46:43.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:46:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:46:43.613+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:46:43.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:46:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:46:43.613+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:46:43.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:46:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:46:43.613+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:46:43.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:46:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:46:43.613+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:46:43.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:46:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:46:43.551+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:46:43.821 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:46:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:46:43.551+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:46:43.821 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:46:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:46:43.551+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:46:43.821 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:46:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:46:43.552+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:46:43.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:46:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:46:43.677+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:46:43.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:46:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:46:43.677+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:46:43.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:46:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:46:43.677+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:46:43.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:46:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:46:43.677+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:46:45.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:46:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:46:44.633+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:46:45.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:46:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:46:44.633+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:46:45.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:46:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:46:44.633+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:46:45.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:46:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:46:44.634+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:46:45.071 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:46:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:46:44.582+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:46:45.071 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:46:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:46:44.582+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:46:45.071 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:46:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:46:44.582+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:46:45.071 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:46:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:46:44.584+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:46:45.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:46:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:46:44.665+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:46:45.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:46:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:46:44.665+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:46:45.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:46:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:46:44.665+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:46:45.072 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:46:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:46:44.665+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:46:45.819 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:46:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:46:45.541+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 23924 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:46:45.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:46:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:46:45.592+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:46:45.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:46:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:46:45.592+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:46:45.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:46:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:46:45.592+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:46:45.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:46:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:46:45.593+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:46:45.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:46:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:46:45.699+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:46:45.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:46:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:46:45.699+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:46:45.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:46:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:46:45.699+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:46:45.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:46:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:46:45.700+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:46:45.821 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:46:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:46:45.537+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:46:45.821 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:46:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:46:45.537+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:46:45.822 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:46:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:46:45.537+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:46:45.822 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:46:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:46:45.538+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:46:46.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:46:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:46:46.723+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:46:46.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:46:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:46:46.723+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:46:46.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:46:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:46:46.723+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:46:46.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:46:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:46:46.724+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:46:46.821 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:46:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:46:46.533+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:46:46.821 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:46:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:46:46.533+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:46:46.821 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:46:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:46:46.533+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:46:46.821 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:46:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:46:46.534+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:46:46.821 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:46:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:46:46.568+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:46:46.821 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:46:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:46:46.568+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:46:46.821 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:46:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:46:46.568+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:46:46.821 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:46:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:46:46.569+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:46:47.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:46:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:46:47.522+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:46:47.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:46:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:46:47.522+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:46:47.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:46:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:46:47.522+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:46:47.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:46:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:46:47.523+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:46:47.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:46:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:46:47.732+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:46:47.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:46:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:46:47.732+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:46:47.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:46:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:46:47.732+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:46:47.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:46:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:46:47.732+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:46:47.821 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:46:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:46:47.548+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:46:47.821 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:46:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:46:47.548+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:46:47.821 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:46:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:46:47.548+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:46:47.822 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:46:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:46:47.548+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:46:48.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:46:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:46:48.535+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:46:48.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:46:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:46:48.535+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:46:48.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:46:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:46:48.535+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:46:48.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:46:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:46:48.536+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:46:48.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:46:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:46:48.504+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:46:48.821 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:46:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:46:48.504+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:46:48.821 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:46:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:46:48.504+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:46:48.821 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:46:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:46:48.505+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:46:48.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:46:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:46:48.718+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:46:48.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:46:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:46:48.718+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:46:48.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:46:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:46:48.718+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:46:48.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:46:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:46:48.718+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:46:49.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:46:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:46:49.571+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:46:49.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:46:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:46:49.571+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:46:49.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:46:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:46:49.571+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:46:49.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:46:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:46:49.572+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:46:49.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:46:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:46:49.701+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:46:49.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:46:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:46:49.701+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:46:49.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:46:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:46:49.701+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:46:49.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:46:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:46:49.702+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:46:49.821 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:46:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:46:49.457+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:46:49.821 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:46:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:46:49.457+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:46:49.821 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:46:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:46:49.457+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:46:49.821 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:46:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:46:49.457+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:46:50.736 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:46:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:46:50.543+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 24007 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:46:50.737 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:46:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:46:50.593+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:46:50.737 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:46:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:46:50.593+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:46:50.737 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:46:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:46:50.593+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:46:50.737 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:46:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:46:50.594+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:46:50.737 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:46:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:46:50.446+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:46:50.738 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:46:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:46:50.446+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:46:50.738 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:46:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:46:50.446+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:46:50.738 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:46:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:46:50.447+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:46:51.035 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:46:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:46:50.735+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:46:51.035 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:46:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:46:50.735+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:46:51.035 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:46:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:46:50.735+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:46:51.036 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:46:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:46:50.736+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:46:51.725 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:46:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:46:51.445+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:46:51.725 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:46:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:46:51.445+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:46:51.725 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:46:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:46:51.445+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:46:51.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:46:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:46:51.445+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:46:51.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:46:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:46:51.570+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:46:51.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:46:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:46:51.570+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:46:51.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:46:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:46:51.570+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:46:51.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:46:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:46:51.570+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:46:52.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:46:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:46:51.771+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:46:52.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:46:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:46:51.771+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:46:52.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:46:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:46:51.771+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:46:52.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:46:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:46:51.772+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:46:52.786 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:46:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:46:52.433+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:46:52.786 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:46:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:46:52.433+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:46:52.786 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:46:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:46:52.433+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:46:52.786 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:46:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:46:52.434+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:46:52.787 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:46:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:46:52.570+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:46:52.787 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:46:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:46:52.570+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:46:52.787 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:46:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:46:52.570+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:46:52.787 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:46:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:46:52.570+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:46:53.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:46:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:46:52.784+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:46:53.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:46:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:46:52.784+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:46:53.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:46:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:46:52.784+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:46:53.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:46:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:46:52.785+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:46:53.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:46:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:46:53.611+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:46:53.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:46:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:46:53.611+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:46:53.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:46:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:46:53.611+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:46:53.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:46:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:46:53.612+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:46:53.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:46:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:46:53.403+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:46:53.821 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:46:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:46:53.403+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:46:53.821 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:46:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:46:53.403+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:46:53.821 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:46:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:46:53.404+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:46:54.107 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:46:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:46:53.827+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:46:54.107 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:46:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:46:53.827+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:46:54.107 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:46:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:46:53.827+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:46:54.108 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:46:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:46:53.828+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:46:54.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:46:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:46:54.625+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:46:54.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:46:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:46:54.625+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:46:54.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:46:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:46:54.625+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:46:54.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:46:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:46:54.626+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:46:54.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:46:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:46:54.433+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:46:54.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:46:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:46:54.433+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:46:54.821 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:46:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:46:54.433+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:46:54.821 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:46:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:46:54.433+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:46:55.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:46:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:46:54.835+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:46:55.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:46:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:46:54.835+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:46:55.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:46:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:46:54.835+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:46:55.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:46:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:46:54.835+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:46:55.820 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:46:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:46:55.546+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 24090 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:46:55.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:46:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:46:55.610+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:46:55.821 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:46:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:46:55.610+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:46:55.821 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:46:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:46:55.610+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:46:55.821 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:46:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:46:55.611+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:46:55.821 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:46:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:46:55.437+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:46:55.821 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:46:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:46:55.437+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:46:55.821 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:46:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:46:55.437+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:46:55.821 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:46:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:46:55.438+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:46:56.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:46:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:46:55.867+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:46:56.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:46:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:46:55.867+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:46:56.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:46:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:46:55.867+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:46:56.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:46:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:46:55.868+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:46:56.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:46:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:46:56.484+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:46:56.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:46:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:46:56.484+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:46:56.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:46:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:46:56.484+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:46:56.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:46:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:46:56.484+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:46:56.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:46:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:46:56.580+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:46:56.821 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:46:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:46:56.580+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:46:56.821 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:46:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:46:56.580+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:46:56.821 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:46:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:46:56.580+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:46:57.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:46:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:46:56.895+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:46:57.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:46:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:46:56.895+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:46:57.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:46:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:46:56.895+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:46:57.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:46:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:46:56.896+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:46:57.821 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:46:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:46:57.594+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:46:57.821 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:46:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:46:57.595+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:46:57.822 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:46:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:46:57.595+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:46:57.822 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:46:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:46:57.595+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:46:57.822 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:46:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:46:57.507+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:46:57.822 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:46:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:46:57.507+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:46:57.822 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:46:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:46:57.507+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:46:57.823 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:46:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:46:57.508+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:46:58.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:46:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:46:57.878+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:46:58.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:46:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:46:57.878+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:46:58.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:46:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:46:57.878+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:46:58.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:46:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:46:57.878+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:46:58.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:46:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:46:58.612+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:46:58.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:46:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:46:58.612+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:46:58.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:46:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:46:58.612+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:46:58.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:46:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:46:58.612+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:46:58.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:46:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:46:58.483+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:46:58.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:46:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:46:58.483+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:46:58.821 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:46:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:46:58.483+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:46:58.821 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:46:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:46:58.484+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:46:59.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:46:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:46:58.881+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:46:59.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:46:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:46:58.881+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:46:59.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:46:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:46:58.881+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:46:59.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:46:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:46:58.882+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:46:59.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:46:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:46:59.644+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:46:59.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:46:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:46:59.644+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:46:59.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:46:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:46:59.644+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:46:59.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:46:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:46:59.645+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:46:59.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:46:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:46:59.516+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:46:59.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:46:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:46:59.516+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:46:59.821 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:46:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:46:59.516+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:46:59.821 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:46:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:46:59.517+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:47:00.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:46:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:46:59.859+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:47:00.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:46:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:46:59.859+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:47:00.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:46:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:46:59.859+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:47:00.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:46:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:46:59.859+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:47:00.819 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:47:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:47:00.548+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 24173 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:47:00.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:47:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:47:00.531+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:47:00.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:47:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:47:00.531+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:47:00.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:47:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:47:00.531+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:47:00.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:47:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:47:00.532+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:47:00.821 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:47:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:47:00.678+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:47:00.821 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:47:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:47:00.678+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:47:00.821 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:47:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:47:00.678+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:47:00.821 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:47:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:47:00.678+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:47:01.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:47:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:47:00.888+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:47:01.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:47:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:47:00.888+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:47:01.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:47:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:47:00.888+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:47:01.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:47:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:47:00.888+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:47:01.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:47:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:47:01.545+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:47:01.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:47:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:47:01.545+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:47:01.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:47:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:47:01.545+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:47:01.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:47:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:47:01.546+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:47:01.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:47:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:47:01.640+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:47:01.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:47:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:47:01.640+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:47:01.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:47:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:47:01.640+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:47:01.821 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:47:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:47:01.641+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:47:02.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:47:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:47:01.918+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:47:02.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:47:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:47:01.918+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:47:02.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:47:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:47:01.918+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:47:02.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:47:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:47:01.919+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:47:02.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:47:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:47:02.596+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:47:02.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:47:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:47:02.596+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:47:02.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:47:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:47:02.596+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:47:02.821 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:47:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:47:02.597+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:47:02.821 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:47:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:47:02.567+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:47:02.821 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:47:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:47:02.567+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:47:02.822 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:47:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:47:02.567+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:47:02.822 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:47:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:47:02.568+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:47:03.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:47:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:47:02.915+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:47:03.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:47:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:47:02.915+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:47:03.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:47:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:47:02.915+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:47:03.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:47:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:47:02.915+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:47:03.626 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:47:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:47:03.555+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:47:03.626 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:47:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:47:03.555+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:47:03.626 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:47:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:47:03.555+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:47:03.626 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:47:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:47:03.556+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:47:03.925 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:47:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:47:03.624+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:47:03.925 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:47:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:47:03.624+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:47:03.926 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:47:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:47:03.624+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:47:03.926 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:47:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:47:03.625+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:47:04.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:47:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:47:03.924+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:47:04.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:47:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:47:03.924+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:47:04.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:47:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:47:03.924+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:47:04.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:47:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:47:03.924+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:47:04.898 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:47:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:47:04.604+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:47:04.898 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:47:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:47:04.604+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:47:04.898 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:47:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:47:04.604+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:47:04.898 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:47:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:47:04.604+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:47:04.898 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:47:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:47:04.635+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:47:04.899 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:47:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:47:04.635+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:47:04.899 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:47:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:47:04.635+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:47:04.899 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:47:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:47:04.636+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:47:05.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:47:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:47:04.896+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:47:05.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:47:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:47:04.896+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:47:05.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:47:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:47:04.896+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:47:05.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:47:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:47:04.897+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:47:05.819 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:47:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:47:05.551+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 24256 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:47:05.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:47:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:47:05.643+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:47:05.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:47:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:47:05.643+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:47:05.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:47:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:47:05.643+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:47:05.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:47:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:47:05.643+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:47:05.821 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:47:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:47:05.616+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:47:05.821 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:47:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:47:05.616+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:47:05.821 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:47:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:47:05.616+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:47:05.821 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:47:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:47:05.617+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:47:06.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:47:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:47:05.860+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:47:06.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:47:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:47:05.860+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:47:06.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:47:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:47:05.860+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:47:06.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:47:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:47:05.861+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:47:06.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:47:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:47:06.568+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:47:06.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:47:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:47:06.568+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:47:06.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:47:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:47:06.568+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:47:06.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:47:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:47:06.569+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:47:06.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:47:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:47:06.647+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:47:06.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:47:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:47:06.647+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:47:06.821 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:47:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:47:06.647+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:47:06.821 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:47:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:47:06.647+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:47:07.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:47:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:47:06.902+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:47:07.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:47:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:47:06.902+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:47:07.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:47:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:47:06.902+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:47:07.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:47:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:47:06.902+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:47:07.926 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:47:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:47:07.667+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:47:07.926 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:47:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:47:07.667+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:47:07.926 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:47:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:47:07.667+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:47:07.926 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:47:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:47:07.668+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:47:07.927 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:47:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:47:07.598+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:47:07.927 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:47:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:47:07.598+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:47:07.927 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:47:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:47:07.598+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:47:07.927 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:47:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:47:07.599+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:47:08.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:47:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:47:07.924+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:47:08.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:47:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:47:07.924+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:47:08.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:47:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:47:07.924+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:47:08.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:47:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:47:07.925+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:47:08.914 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:47:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:47:08.626+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:47:08.914 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:47:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:47:08.626+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:47:08.915 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:47:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:47:08.626+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:47:08.915 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:47:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:47:08.627+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:47:08.915 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:47:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:47:08.582+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:47:08.915 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:47:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:47:08.582+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:47:08.915 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:47:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:47:08.582+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:47:08.916 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:47:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:47:08.583+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:47:09.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:47:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:47:08.913+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:47:09.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:47:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:47:08.913+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:47:09.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:47:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:47:08.913+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:47:09.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:47:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:47:08.913+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:47:09.921 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:47:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:47:09.645+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:47:09.921 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:47:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:47:09.645+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:47:09.921 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:47:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:47:09.645+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:47:09.921 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:47:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:47:09.646+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:47:09.922 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:47:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:47:09.571+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:47:09.922 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:47:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:47:09.571+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:47:09.922 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:47:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:47:09.571+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:47:09.922 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:47:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:47:09.571+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:47:10.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:47:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:47:09.919+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:47:10.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:47:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:47:09.919+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:47:10.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:47:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:47:09.919+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:47:10.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:47:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:47:09.920+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:47:10.820 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:47:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:47:10.553+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 24339 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:47:10.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:47:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:47:10.608+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:47:10.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:47:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:47:10.608+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:47:10.821 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:47:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:47:10.608+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:47:10.821 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:47:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:47:10.609+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:47:10.821 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:47:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:47:10.554+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:47:10.821 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:47:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:47:10.554+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:47:10.821 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:47:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:47:10.554+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:47:10.821 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:47:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:47:10.556+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:47:11.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:47:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:47:10.902+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:47:11.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:47:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:47:10.902+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:47:11.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:47:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:47:10.902+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:47:11.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:47:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:47:10.902+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:47:11.945 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:47:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:47:11.582+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:47:11.945 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:47:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:47:11.582+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:47:11.945 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:47:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:47:11.582+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:47:11.945 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:47:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:47:11.583+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:47:11.946 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:47:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:47:11.592+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:47:11.946 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:47:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:47:11.592+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:47:11.946 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:47:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:47:11.592+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:47:11.946 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:47:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:47:11.592+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:47:12.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:47:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:47:11.943+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:47:12.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:47:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:47:11.943+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:47:12.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:47:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:47:11.943+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:47:12.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:47:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:47:11.944+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:47:12.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:47:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:47:12.590+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:47:12.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:47:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:47:12.590+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:47:12.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:47:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:47:12.590+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:47:12.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:47:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:47:12.591+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:47:12.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:47:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:47:12.604+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:47:12.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:47:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:47:12.604+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:47:12.821 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:47:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:47:12.604+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:47:12.821 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:47:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:47:12.605+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:47:13.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:47:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:47:12.904+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:47:13.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:47:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:47:12.904+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:47:13.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:47:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:47:12.904+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:47:13.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:47:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:47:12.905+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:47:13.940 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:47:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:47:13.651+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:47:13.941 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:47:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:47:13.651+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:47:13.941 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:47:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:47:13.651+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:47:13.941 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:47:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:47:13.652+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:47:13.942 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:47:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:47:13.633+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:47:13.942 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:47:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:47:13.633+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:47:13.942 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:47:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:47:13.633+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:47:13.942 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:47:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:47:13.634+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:47:14.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:47:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:47:13.939+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:47:14.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:47:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:47:13.939+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:47:14.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:47:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:47:13.939+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:47:14.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:47:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:47:13.940+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:47:14.985 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:47:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:47:14.624+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:47:14.985 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:47:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:47:14.624+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:47:14.985 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:47:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:47:14.624+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:47:14.985 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:47:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:47:14.624+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:47:14.986 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:47:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:47:14.626+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:47:14.986 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:47:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:47:14.626+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:47:14.986 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:47:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:47:14.626+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:47:14.986 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:47:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:47:14.627+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:47:15.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:47:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:47:14.983+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:47:15.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:47:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:47:14.983+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:47:15.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:47:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:47:14.983+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:47:15.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:47:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:47:14.984+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:47:15.820 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:47:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:47:15.556+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 24422 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:47:15.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:47:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:47:15.600+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:47:15.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:47:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:47:15.600+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:47:15.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:47:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:47:15.600+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:47:15.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:47:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:47:15.601+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:47:15.821 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:47:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:47:15.662+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:47:15.821 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:47:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:47:15.662+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:47:15.821 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:47:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:47:15.662+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:47:15.821 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:47:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:47:15.663+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:47:16.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:47:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:47:15.945+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:47:16.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:47:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:47:15.945+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:47:16.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:47:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:47:15.945+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:47:16.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:47:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:47:15.945+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:47:16.929 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:47:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:47:16.702+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:47:16.930 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:47:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:47:16.702+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:47:16.930 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:47:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:47:16.702+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:47:16.930 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:47:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:47:16.702+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:47:16.930 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:47:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:47:16.643+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:47:16.931 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:47:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:47:16.643+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:47:16.931 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:47:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:47:16.643+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:47:16.931 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:47:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:47:16.644+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:47:17.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:47:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:47:16.928+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:47:17.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:47:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:47:16.928+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:47:17.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:47:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:47:16.928+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:47:17.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:47:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:47:16.928+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:47:18.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:47:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:47:17.738+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:47:18.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:47:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:47:17.738+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:47:18.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:47:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:47:17.738+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:47:18.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:47:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:47:17.739+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:47:18.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:47:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:47:17.651+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:47:18.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:47:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:47:17.651+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:47:18.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:47:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:47:17.651+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:47:18.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:47:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:47:17.652+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:47:18.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:47:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:47:17.897+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:47:18.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:47:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:47:17.897+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:47:18.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:47:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:47:17.897+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:47:18.072 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:47:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:47:17.898+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:47:19.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:47:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:47:18.747+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:47:19.071 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:47:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:47:18.747+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:47:19.071 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:47:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:47:18.747+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:47:19.071 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:47:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:47:18.748+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:47:19.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:47:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:47:18.855+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:47:19.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:47:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:47:18.855+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:47:19.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:47:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:47:18.855+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:47:19.072 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:47:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:47:18.856+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:47:19.072 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:47:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:47:18.609+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:47:19.072 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:47:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:47:18.609+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:47:19.072 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:47:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:47:18.609+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:47:19.072 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:47:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:47:18.610+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:47:20.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:47:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:47:19.737+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:47:20.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:47:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:47:19.737+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:47:20.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:47:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:47:19.737+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:47:20.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:47:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:47:19.738+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:47:20.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:47:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:47:19.648+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:47:20.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:47:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:47:19.648+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:47:20.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:47:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:47:19.648+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:47:20.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:47:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:47:19.649+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:47:20.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:47:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:47:19.819+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:47:20.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:47:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:47:19.819+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:47:20.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:47:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:47:19.819+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:47:20.072 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:47:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:47:19.819+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:47:20.820 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:47:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:47:20.559+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 24505 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:47:20.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:47:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:47:20.788+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:47:20.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:47:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:47:20.788+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:47:20.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:47:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:47:20.788+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:47:20.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:47:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:47:20.789+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:47:20.821 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:47:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:47:20.685+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:47:20.821 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:47:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:47:20.685+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:47:20.821 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:47:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:47:20.685+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:47:20.821 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:47:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:47:20.685+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:47:20.822 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:47:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:47:20.734+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:47:20.822 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:47:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:47:20.734+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:47:20.822 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:47:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:47:20.734+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:47:20.822 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:47:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:47:20.735+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:47:22.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:47:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:47:21.697+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:47:22.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:47:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:47:21.697+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:47:22.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:47:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:47:21.697+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:47:22.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:47:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:47:21.699+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:47:22.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:47:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:47:21.668+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:47:22.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:47:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:47:21.668+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:47:22.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:47:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:47:21.668+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:47:22.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:47:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:47:21.669+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:47:22.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:47:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:47:21.823+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:47:22.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:47:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:47:21.823+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:47:22.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:47:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:47:21.823+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:47:22.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:47:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:47:21.824+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:47:23.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:47:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:47:22.695+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:47:23.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:47:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:47:22.695+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:47:23.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:47:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:47:22.695+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:47:23.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:47:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:47:22.696+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:47:23.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:47:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:47:22.683+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:47:23.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:47:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:47:22.683+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:47:23.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:47:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:47:22.683+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:47:23.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:47:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:47:22.684+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:47:23.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:47:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:47:22.782+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:47:23.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:47:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:47:22.782+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:47:23.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:47:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:47:22.782+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:47:23.072 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:47:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:47:22.782+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:47:24.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:47:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:47:23.681+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:47:24.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:47:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:47:23.681+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:47:24.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:47:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:47:23.681+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:47:24.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:47:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:47:23.682+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:47:24.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:47:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:47:23.687+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:47:24.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:47:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:47:23.687+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:47:24.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:47:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:47:23.687+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:47:24.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:47:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:47:23.688+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:47:24.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:47:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:47:23.825+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:47:24.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:47:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:47:23.825+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:47:24.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:47:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:47:23.825+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:47:24.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:47:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:47:23.826+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:47:25.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:47:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:47:24.687+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:47:25.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:47:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:47:24.687+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:47:25.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:47:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:47:24.687+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:47:25.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:47:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:47:24.688+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:47:25.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:47:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:47:24.634+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:47:25.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:47:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:47:24.634+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:47:25.071 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:47:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:47:24.634+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:47:25.071 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:47:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:47:24.635+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:47:25.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:47:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:47:24.865+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:47:25.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:47:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:47:24.865+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:47:25.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:47:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:47:24.865+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:47:25.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:47:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:47:24.865+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:47:25.819 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:47:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:47:25.561+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 24588 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:47:25.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:47:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:47:25.668+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:47:25.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:47:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:47:25.668+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:47:25.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:47:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:47:25.668+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:47:25.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:47:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:47:25.669+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:47:25.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:47:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:47:25.723+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:47:25.821 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:47:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:47:25.723+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:47:25.821 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:47:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:47:25.723+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:47:25.821 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:47:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:47:25.724+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:47:26.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:47:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:47:25.887+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:47:26.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:47:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:47:25.887+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:47:26.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:47:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:47:25.887+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:47:26.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:47:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:47:25.887+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:47:27.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:47:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:47:26.872+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:47:27.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:47:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:47:26.872+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:47:27.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:47:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:47:26.872+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:47:27.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:47:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:47:26.872+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:47:27.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:47:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:47:26.728+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:47:27.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:47:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:47:26.728+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:47:27.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:47:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:47:26.728+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:47:27.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:47:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:47:26.728+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:47:27.071 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:47:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:47:26.643+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:47:27.071 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:47:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:47:26.643+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:47:27.071 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:47:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:47:26.643+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:47:27.071 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:47:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:47:26.644+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:47:28.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:47:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:47:27.830+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:47:28.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:47:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:47:27.830+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:47:28.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:47:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:47:27.830+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:47:28.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:47:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:47:27.830+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:47:28.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:47:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:47:27.670+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:47:28.071 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:47:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:47:27.670+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:47:28.071 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:47:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:47:27.670+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:47:28.071 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:47:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:47:27.671+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:47:28.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:47:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:47:27.778+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:47:28.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:47:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:47:27.778+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:47:28.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:47:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:47:27.778+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:47:28.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:47:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:47:27.779+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:47:29.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:47:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:47:28.826+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:47:29.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:47:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:47:28.826+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:47:29.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:47:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:47:28.826+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:47:29.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:47:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:47:28.827+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:47:29.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:47:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:47:28.630+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:47:29.071 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:47:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:47:28.630+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:47:29.071 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:47:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:47:28.630+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:47:29.071 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:47:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:47:28.631+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:47:29.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:47:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:47:28.821+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:47:29.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:47:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:47:28.821+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:47:29.072 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:47:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:47:28.821+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:47:29.072 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:47:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:47:28.822+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:47:30.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:47:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:47:29.875+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:47:30.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:47:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:47:29.875+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:47:30.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:47:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:47:29.875+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:47:30.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:47:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:47:29.876+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:47:30.071 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:47:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:47:29.670+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:47:30.071 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:47:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:47:29.670+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:47:30.071 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:47:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:47:29.670+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:47:30.071 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:47:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:47:29.671+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:47:30.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:47:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:47:29.862+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:47:30.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:47:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:47:29.862+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:47:30.072 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:47:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:47:29.862+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:47:30.072 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:47:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:47:29.863+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:47:30.819 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:47:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:47:30.563+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 24671 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:47:30.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:47:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:47:30.684+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:47:30.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:47:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:47:30.684+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:47:30.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:47:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:47:30.684+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:47:30.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:47:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:47:30.684+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:47:31.319 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:47:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:47:30.832+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:47:31.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:47:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:47:30.832+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:47:31.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:47:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:47:30.832+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:47:31.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:47:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:47:30.833+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:47:31.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:47:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:47:30.880+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:47:31.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:47:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:47:30.880+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:47:31.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:47:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:47:30.880+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:47:31.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:47:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:47:30.881+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:47:32.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:47:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:47:31.730+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:47:32.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:47:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:47:31.730+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:47:32.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:47:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:47:31.730+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:47:32.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:47:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:47:31.732+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:47:32.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:47:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:47:31.855+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:47:32.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:47:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:47:31.855+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:47:32.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:47:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:47:31.855+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:47:32.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:47:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:47:31.856+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:47:32.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:47:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:47:31.928+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:47:32.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:47:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:47:31.928+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:47:32.072 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:47:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:47:31.928+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:47:32.072 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:47:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:47:31.928+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:47:33.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:47:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:47:32.752+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:47:33.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:47:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:47:32.752+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:47:33.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:47:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:47:32.752+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:47:33.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:47:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:47:32.753+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:47:33.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:47:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:47:32.911+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:47:33.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:47:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:47:32.911+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:47:33.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:47:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:47:32.911+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:47:33.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:47:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:47:32.911+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:47:33.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:47:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:47:32.884+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:47:33.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:47:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:47:32.884+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:47:33.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:47:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:47:32.884+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:47:33.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:47:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:47:32.885+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:47:34.046 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:47:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:47:33.850+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:47:34.046 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:47:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:47:33.850+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:47:34.046 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:47:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:47:33.850+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:47:34.046 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:47:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:47:33.850+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:47:34.047 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:47:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:47:33.912+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:47:34.047 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:47:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:47:33.912+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:47:34.047 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:47:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:47:33.912+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:47:34.047 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:47:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:47:33.913+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:47:34.047 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:47:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:47:33.722+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:47:34.047 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:47:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:47:33.722+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:47:34.048 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:47:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:47:33.722+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:47:34.048 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:47:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:47:33.723+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:47:35.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:47:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:47:34.722+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:47:35.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:47:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:47:34.722+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:47:35.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:47:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:47:34.722+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:47:35.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:47:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:47:34.722+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:47:35.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:47:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:47:34.900+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:47:35.073 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:47:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:47:34.900+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:47:35.073 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:47:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:47:34.900+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:47:35.073 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:47:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:47:34.901+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:47:35.074 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:47:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:47:34.950+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:47:35.074 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:47:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:47:34.950+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:47:35.074 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:47:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:47:34.950+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:47:35.074 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:47:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:47:34.950+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:47:35.820 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:47:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:47:35.565+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 24754 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:47:35.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:47:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:47:35.719+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:47:35.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:47:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:47:35.719+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:47:35.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:47:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:47:35.719+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:47:35.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:47:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:47:35.720+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:47:36.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:47:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:47:35.930+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:47:36.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:47:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:47:35.930+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:47:36.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:47:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:47:35.930+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:47:36.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:47:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:47:35.931+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:47:36.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:47:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:47:35.961+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:47:36.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:47:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:47:35.961+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:47:36.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:47:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:47:35.961+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:47:36.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:47:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:47:35.962+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:47:37.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:47:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:47:36.736+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:47:37.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:47:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:47:36.736+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:47:37.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:47:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:47:36.736+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:47:37.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:47:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:47:36.736+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:47:37.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:47:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:47:36.933+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:47:37.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:47:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:47:36.933+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:47:37.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:47:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:47:36.933+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:47:37.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:47:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:47:36.933+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:47:37.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:47:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:47:36.950+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:47:37.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:47:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:47:36.950+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:47:37.072 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:47:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:47:36.950+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:47:37.072 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:47:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:47:36.950+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:47:38.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:47:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:47:37.737+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:47:38.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:47:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:47:37.737+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:47:38.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:47:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:47:37.737+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:47:38.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:47:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:47:37.737+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:47:38.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:47:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:47:37.937+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:47:38.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:47:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:47:37.937+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:47:38.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:47:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:47:37.937+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:47:38.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:47:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:47:37.938+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:47:38.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:47:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:47:37.967+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:47:38.072 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:47:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:47:37.967+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:47:38.072 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:47:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:47:37.967+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:47:38.072 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:47:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:47:37.967+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:47:38.990 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:47:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:47:38.724+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:47:38.991 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:47:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:47:38.724+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:47:38.991 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:47:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:47:38.724+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:47:38.991 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:47:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:47:38.724+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:47:39.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:47:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:47:38.988+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:47:39.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:47:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:47:38.988+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:47:39.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:47:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:47:38.988+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:47:39.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:47:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:47:38.988+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:47:39.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:47:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:47:38.998+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:47:39.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:47:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:47:38.998+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:47:39.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:47:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:47:38.998+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:47:39.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:47:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:47:39.000+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:47:40.023 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:47:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:47:39.750+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:47:40.023 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:47:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:47:39.750+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:47:40.023 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:47:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:47:39.750+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:47:40.024 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:47:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:47:39.750+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:47:40.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:47:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:47:40.044+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:47:40.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:47:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:47:40.044+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:47:40.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:47:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:47:40.044+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:47:40.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:47:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:47:40.045+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:47:40.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:47:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:47:40.021+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:47:40.321 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:47:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:47:40.021+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:47:40.321 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:47:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:47:40.021+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:47:40.321 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:47:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:47:40.022+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:47:41.049 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:47:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:47:40.569+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 24836 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:47:41.049 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:47:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:47:40.707+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:47:41.049 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:47:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:47:40.707+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:47:41.050 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:47:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:47:40.707+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:47:41.050 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:47:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:47:40.708+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:47:41.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:47:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:47:41.047+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:47:41.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:47:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:47:41.047+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:47:41.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:47:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:47:41.047+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:47:41.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:47:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:47:41.048+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:47:41.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:47:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:47:41.085+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:47:41.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:47:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:47:41.085+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:47:41.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:47:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:47:41.085+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:47:41.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:47:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:47:41.086+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:47:42.046 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:47:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:47:41.703+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:47:42.047 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:47:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:47:41.703+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:47:42.047 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:47:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:47:41.703+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:47:42.047 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:47:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:47:41.704+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:47:42.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:47:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:47:42.045+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:47:42.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:47:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:47:42.045+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:47:42.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:47:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:47:42.045+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:47:42.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:47:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:47:42.046+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:47:42.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:47:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:47:42.074+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:47:42.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:47:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:47:42.074+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:47:42.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:47:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:47:42.074+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:47:42.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:47:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:47:42.074+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:47:43.063 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:47:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:47:42.711+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:47:43.064 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:47:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:47:42.711+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:47:43.064 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:47:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:47:42.711+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:47:43.064 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:47:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:47:42.711+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:47:43.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:47:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:47:43.086+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:47:43.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:47:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:47:43.086+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:47:43.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:47:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:47:43.086+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:47:43.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:47:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:47:43.087+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:47:43.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:47:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:47:43.062+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:47:43.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:47:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:47:43.062+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:47:43.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:47:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:47:43.062+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:47:43.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:47:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:47:43.062+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:47:44.051 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:47:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:47:43.685+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:47:44.052 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:47:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:47:43.685+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:47:44.052 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:47:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:47:43.685+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:47:44.052 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:47:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:47:43.686+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:47:44.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:47:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:47:44.103+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:47:44.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:47:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:47:44.103+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:47:44.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:47:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:47:44.103+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:47:44.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:47:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:47:44.104+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:47:44.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:47:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:47:44.049+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:47:44.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:47:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:47:44.049+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:47:44.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:47:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:47:44.049+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:47:44.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:47:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:47:44.050+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:47:45.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:47:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:47:44.660+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:47:45.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:47:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:47:44.660+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:47:45.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:47:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:47:44.660+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:47:45.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:47:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:47:44.660+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:47:45.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:47:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:47:45.075+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:47:45.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:47:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:47:45.075+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:47:45.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:47:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:47:45.075+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:47:45.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:47:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:47:45.075+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:47:45.571 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:47:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:47:45.137+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:47:45.571 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:47:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:47:45.137+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:47:45.571 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:47:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:47:45.137+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:47:45.571 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:47:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:47:45.138+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:47:46.069 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:47:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:47:45.572+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 24920 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:47:46.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:47:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:47:45.706+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:47:46.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:47:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:47:45.706+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:47:46.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:47:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:47:45.706+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:47:46.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:47:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:47:45.707+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:47:46.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:47:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:47:46.165+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:47:46.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:47:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:47:46.165+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:47:46.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:47:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:47:46.165+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:47:46.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:47:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:47:46.166+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:47:46.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:47:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:47:46.099+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:47:46.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:47:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:47:46.099+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:47:46.571 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:47:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:47:46.099+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:47:46.571 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:47:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:47:46.100+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:47:47.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:47:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:47:46.676+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:47:47.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:47:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:47:46.676+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:47:47.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:47:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:47:46.676+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:47:47.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:47:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:47:46.676+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:47:47.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:47:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:47:47.139+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:47:47.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:47:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:47:47.140+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:47:47.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:47:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:47:47.140+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:47:47.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:47:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:47:47.140+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:47:47.571 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:47:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:47:47.091+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:47:47.571 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:47:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:47:47.091+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:47:47.571 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:47:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:47:47.091+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:47:47.571 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:47:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:47:47.092+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:47:48.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:47:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:47:47.705+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:47:48.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:47:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:47:47.705+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:47:48.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:47:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:47:47.706+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:47:48.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:47:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:47:47.706+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:47:48.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:47:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:47:48.184+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:47:48.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:47:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:47:48.184+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:47:48.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:47:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:47:48.184+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:47:48.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:47:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:47:48.185+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:47:48.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:47:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:47:48.079+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:47:48.571 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:47:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:47:48.079+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:47:48.571 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:47:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:47:48.079+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:47:48.571 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:47:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:47:48.079+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:47:49.068 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:47:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:47:48.677+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:47:49.068 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:47:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:47:48.677+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:47:49.068 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:47:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:47:48.677+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:47:49.068 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:47:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:47:48.678+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:47:49.319 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:47:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:47:49.218+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:47:49.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:47:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:47:49.218+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:47:49.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:47:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:47:49.218+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:47:49.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:47:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:47:49.219+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:47:49.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:47:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:47:49.066+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:47:49.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:47:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:47:49.066+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:47:49.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:47:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:47:49.066+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:47:49.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:47:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:47:49.067+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:47:50.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:47:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:47:49.700+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:47:50.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:47:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:47:49.700+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:47:50.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:47:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:47:49.700+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:47:50.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:47:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:47:49.701+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:47:50.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:47:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:47:50.234+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:47:50.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:47:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:47:50.234+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:47:50.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:47:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:47:50.234+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:47:50.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:47:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:47:50.235+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:47:50.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:47:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:47:50.095+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:47:50.571 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:47:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:47:50.095+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:47:50.571 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:47:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:47:50.095+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:47:50.571 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:47:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:47:50.096+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:47:51.061 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:47:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:47:50.574+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 25002 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:47:51.062 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:47:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:47:50.679+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:47:51.062 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:47:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:47:50.679+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:47:51.062 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:47:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:47:50.679+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:47:51.062 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:47:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:47:50.679+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:47:51.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:47:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:47:51.206+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:47:51.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:47:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:47:51.206+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:47:51.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:47:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:47:51.206+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:47:51.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:47:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:47:51.206+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:47:51.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:47:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:47:51.060+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:47:51.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:47:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:47:51.060+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:47:51.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:47:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:47:51.060+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:47:51.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:47:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:47:51.061+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:47:52.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:47:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:47:51.659+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:47:52.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:47:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:47:51.659+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:47:52.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:47:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:47:51.659+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:47:52.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:47:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:47:51.659+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:47:52.475 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:47:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:47:52.167+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:47:52.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:47:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:47:52.167+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:47:52.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:47:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:47:52.167+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:47:52.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:47:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:47:52.168+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:47:52.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:47:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:47:52.090+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:47:52.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:47:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:47:52.090+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:47:52.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:47:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:47:52.090+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:47:52.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:47:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:47:52.090+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:47:52.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:47:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:47:52.648+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:47:52.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:47:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:47:52.648+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:47:52.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:47:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:47:52.648+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:47:52.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:47:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:47:52.649+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:47:53.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:47:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:47:53.161+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:47:53.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:47:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:47:53.161+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:47:53.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:47:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:47:53.161+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:47:53.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:47:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:47:53.163+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:47:53.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:47:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:47:53.059+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:47:53.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:47:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:47:53.059+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:47:53.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:47:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:47:53.059+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:47:53.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:47:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:47:53.060+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:47:54.044 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:47:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:47:53.678+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:47:54.044 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:47:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:47:53.678+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:47:54.044 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:47:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:47:53.678+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:47:54.045 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:47:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:47:53.678+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:47:54.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:47:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:47:54.152+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:47:54.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:47:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:47:54.152+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:47:54.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:47:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:47:54.152+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:47:54.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:47:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:47:54.152+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:47:54.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:47:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:47:54.043+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:47:54.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:47:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:47:54.043+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:47:54.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:47:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:47:54.043+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:47:54.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:47:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:47:54.043+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:47:55.004 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:47:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:47:54.637+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:47:55.004 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:47:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:47:54.637+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:47:55.004 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:47:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:47:54.637+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:47:55.005 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:47:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:47:54.638+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:47:55.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:47:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:47:55.002+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:47:55.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:47:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:47:55.003+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:47:55.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:47:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:47:55.003+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:47:55.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:47:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:47:55.003+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:47:55.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:47:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:47:55.164+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:47:55.321 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:47:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:47:55.164+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:47:55.321 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:47:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:47:55.164+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:47:55.321 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:47:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:47:55.165+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:47:55.983 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:47:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:47:55.577+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 25086 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:47:55.984 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:47:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:47:55.589+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:47:55.984 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:47:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:47:55.589+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:47:55.984 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:47:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:47:55.589+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:47:55.985 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:47:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:47:55.590+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:47:56.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:47:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:47:56.124+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:47:56.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:47:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:47:56.124+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:47:56.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:47:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:47:56.124+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:47:56.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:47:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:47:56.125+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:47:56.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:47:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:47:55.982+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:47:56.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:47:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:47:55.982+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:47:56.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:47:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:47:55.982+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:47:56.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:47:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:47:55.982+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:47:56.988 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:47:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:47:56.638+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:47:56.988 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:47:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:47:56.638+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:47:56.988 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:47:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:47:56.638+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:47:56.989 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:47:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:47:56.639+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:47:57.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:47:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:47:56.986+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:47:57.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:47:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:47:56.986+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:47:57.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:47:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:47:56.986+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:47:57.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:47:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:47:56.987+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:47:57.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:47:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:47:57.158+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:47:57.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:47:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:47:57.158+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:47:57.321 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:47:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:47:57.158+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:47:57.321 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:47:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:47:57.159+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:47:57.983 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:47:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:47:57.639+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:47:57.983 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:47:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:47:57.639+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:47:57.983 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:47:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:47:57.639+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:47:57.984 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:47:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:47:57.640+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:47:58.319 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:47:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:47:58.141+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:47:58.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:47:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:47:58.141+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:47:58.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:47:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:47:58.141+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:47:58.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:47:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:47:58.142+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:47:58.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:47:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:47:57.981+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:47:58.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:47:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:47:57.981+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:47:58.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:47:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:47:57.981+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:47:58.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:47:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:47:57.981+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:47:59.024 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:47:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:47:58.594+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:47:59.025 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:47:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:47:58.594+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:47:59.025 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:47:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:47:58.594+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:47:59.025 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:47:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:47:58.595+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:47:59.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:47:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:47:59.117+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:47:59.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:47:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:47:59.117+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:47:59.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:47:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:47:59.117+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:47:59.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:47:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:47:59.118+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:47:59.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:47:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:47:59.023+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:47:59.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:47:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:47:59.023+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:47:59.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:47:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:47:59.023+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:47:59.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:47:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:47:59.023+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:47:59.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:47:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:47:59.555+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:47:59.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:47:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:47:59.555+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:47:59.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:47:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:47:59.555+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:47:59.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:47:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:47:59.556+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:48:00.319 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:48:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:48:00.108+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:48:00.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:48:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:48:00.108+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:48:00.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:48:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:48:00.108+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:48:00.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:48:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:48:00.109+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:48:00.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:48:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:48:00.057+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:48:00.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:48:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:48:00.057+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:48:00.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:48:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:48:00.057+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:48:00.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:48:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:48:00.057+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:48:00.819 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:48:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:48:00.579+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 25168 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:48:00.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:48:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:48:00.558+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:48:00.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:48:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:48:00.558+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:48:00.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:48:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:48:00.558+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:48:00.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:48:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:48:00.559+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:48:01.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:48:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:48:01.017+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:48:01.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:48:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:48:01.017+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:48:01.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:48:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:48:01.017+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:48:01.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:48:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:48:01.018+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:48:01.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:48:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:48:01.100+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:48:01.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:48:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:48:01.100+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:48:01.321 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:48:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:48:01.100+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:48:01.321 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:48:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:48:01.101+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:48:02.020 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:48:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:48:01.605+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:48:02.020 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:48:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:48:01.605+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:48:02.020 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:48:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:48:01.605+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:48:02.020 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:48:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:48:01.606+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:48:02.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:48:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:48:02.089+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:48:02.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:48:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:48:02.089+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:48:02.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:48:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:48:02.089+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:48:02.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:48:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:48:02.090+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:48:02.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:48:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:48:02.018+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:48:02.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:48:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:48:02.018+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:48:02.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:48:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:48:02.018+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:48:02.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:48:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:48:02.019+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:48:02.982 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:48:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:48:02.628+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:48:02.982 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:48:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:48:02.628+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:48:02.982 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:48:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:48:02.628+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:48:02.982 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:48:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:48:02.629+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:48:03.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:48:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:48:03.045+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:48:03.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:48:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:48:03.045+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:48:03.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:48:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:48:03.045+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:48:03.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:48:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:48:03.046+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:48:03.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:48:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:48:02.980+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:48:03.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:48:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:48:02.980+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:48:03.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:48:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:48:02.980+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:48:03.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:48:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:48:02.981+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:48:03.971 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:48:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:48:03.622+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:48:03.971 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:48:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:48:03.622+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:48:03.971 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:48:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:48:03.622+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:48:03.972 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:48:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:48:03.623+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:48:04.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:48:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:48:04.045+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:48:04.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:48:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:48:04.045+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:48:04.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:48:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:48:04.045+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:48:04.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:48:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:48:04.046+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:48:04.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:48:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:48:03.970+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:48:04.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:48:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:48:03.970+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:48:04.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:48:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:48:03.970+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:48:04.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:48:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:48:03.970+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:48:05.007 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:48:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:48:04.667+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:48:05.007 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:48:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:48:04.667+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:48:05.007 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:48:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:48:04.667+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:48:05.007 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:48:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:48:04.668+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:48:05.319 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:48:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:48:05.019+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:48:05.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:48:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:48:05.019+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:48:05.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:48:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:48:05.019+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:48:05.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:48:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:48:05.019+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:48:05.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:48:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:48:05.005+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:48:05.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:48:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:48:05.005+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:48:05.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:48:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:48:05.005+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:48:05.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:48:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:48:05.006+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:48:05.985 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:48:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:48:05.582+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 25252 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:48:05.986 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:48:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:48:05.633+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:48:05.986 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:48:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:48:05.633+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:48:05.986 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:48:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:48:05.633+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:48:05.986 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:48:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:48:05.634+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:48:06.319 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:48:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:48:06.038+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:48:06.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:48:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:48:06.038+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:48:06.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:48:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:48:06.038+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:48:06.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:48:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:48:06.039+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:48:06.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:48:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:48:05.982+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:48:06.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:48:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:48:05.982+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:48:06.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:48:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:48:05.982+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:48:06.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:48:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:48:05.983+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:48:07.000 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:48:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:48:06.623+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:48:07.001 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:48:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:48:06.623+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:48:07.001 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:48:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:48:06.623+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:48:07.001 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:48:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:48:06.624+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:48:07.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:48:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:48:07.062+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:48:07.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:48:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:48:07.062+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:48:07.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:48:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:48:07.062+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:48:07.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:48:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:48:07.063+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:48:07.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:48:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:48:06.999+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:48:07.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:48:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:48:06.999+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:48:07.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:48:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:48:06.999+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:48:07.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:48:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:48:06.999+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:48:08.019 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:48:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:48:07.618+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:48:08.020 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:48:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:48:07.618+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:48:08.020 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:48:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:48:07.618+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:48:08.020 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:48:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:48:07.619+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:48:08.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:48:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:48:08.053+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:48:08.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:48:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:48:08.053+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:48:08.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:48:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:48:08.053+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:48:08.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:48:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:48:08.053+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:48:08.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:48:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:48:08.018+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:48:08.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:48:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:48:08.018+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:48:08.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:48:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:48:08.018+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:48:08.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:48:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:48:08.019+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:48:08.974 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:48:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:48:08.665+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:48:08.974 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:48:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:48:08.665+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:48:08.974 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:48:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:48:08.665+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:48:08.975 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:48:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:48:08.666+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:48:09.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:48:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:48:09.090+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:48:09.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:48:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:48:09.090+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:48:09.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:48:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:48:09.090+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:48:09.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:48:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:48:09.091+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:48:09.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:48:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:48:08.973+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:48:09.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:48:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:48:08.973+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:48:09.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:48:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:48:08.973+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:48:09.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:48:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:48:08.973+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:48:09.973 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:48:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:48:09.651+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:48:09.973 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:48:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:48:09.651+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:48:09.973 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:48:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:48:09.651+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:48:09.974 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:48:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:48:09.653+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:48:10.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:48:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:48:10.105+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:48:10.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:48:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:48:10.105+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:48:10.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:48:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:48:10.105+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:48:10.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:48:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:48:10.106+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:48:10.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:48:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:48:09.972+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:48:10.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:48:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:48:09.972+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:48:10.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:48:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:48:09.972+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:48:10.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:48:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:48:09.972+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:48:10.907 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:48:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:48:10.585+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 25334 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:48:10.908 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:48:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:48:10.629+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:48:10.908 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:48:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:48:10.629+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:48:10.908 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:48:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:48:10.629+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:48:10.908 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:48:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:48:10.630+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:48:11.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:48:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:48:10.959+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:48:11.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:48:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:48:10.959+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:48:11.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:48:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:48:10.959+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:48:11.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:48:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:48:10.960+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:48:11.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:48:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:48:11.120+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:48:11.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:48:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:48:11.120+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:48:11.321 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:48:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:48:11.120+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:48:11.321 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:48:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:48:11.120+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:48:11.974 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:48:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:48:11.673+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:48:11.974 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:48:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:48:11.673+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:48:11.974 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:48:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:48:11.673+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:48:11.974 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:48:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:48:11.675+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:48:12.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:48:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:48:12.079+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:48:12.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:48:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:48:12.079+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:48:12.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:48:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:48:12.079+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:48:12.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:48:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:48:12.079+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:48:12.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:48:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:48:11.972+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:48:12.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:48:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:48:11.972+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:48:12.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:48:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:48:11.972+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:48:12.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:48:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:48:11.973+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:48:13.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:48:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:48:12.711+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:48:13.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:48:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:48:12.711+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:48:13.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:48:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:48:12.711+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:48:13.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:48:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:48:12.712+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:48:13.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:48:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:48:12.937+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:48:13.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:48:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:48:12.937+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:48:13.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:48:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:48:12.937+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:48:13.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:48:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:48:12.937+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:48:13.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:48:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:48:13.095+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:48:13.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:48:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:48:13.095+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:48:13.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:48:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:48:13.095+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:48:13.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:48:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:48:13.095+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:48:13.979 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:48:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:48:13.910+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:48:13.979 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:48:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:48:13.910+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:48:13.980 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:48:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:48:13.910+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:48:13.980 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:48:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:48:13.911+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:48:13.980 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:48:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:48:13.689+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:48:13.980 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:48:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:48:13.689+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:48:13.980 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:48:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:48:13.689+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:48:13.980 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:48:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:48:13.690+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:48:14.319 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:48:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:48:14.112+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:48:14.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:48:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:48:14.112+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:48:14.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:48:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:48:14.112+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:48:14.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:48:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:48:14.112+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:48:14.960 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:48:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:48:14.672+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:48:14.960 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:48:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:48:14.672+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:48:14.960 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:48:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:48:14.672+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:48:14.960 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:48:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:48:14.673+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:48:15.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:48:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:48:14.958+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:48:15.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:48:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:48:14.958+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:48:15.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:48:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:48:14.958+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:48:15.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:48:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:48:14.959+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:48:15.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:48:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:48:15.087+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:48:15.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:48:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:48:15.087+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:48:15.321 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:48:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:48:15.087+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:48:15.321 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:48:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:48:15.088+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:48:15.983 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:48:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:48:15.588+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 25418 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:48:15.983 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:48:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:48:15.678+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:48:15.984 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:48:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:48:15.678+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:48:15.984 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:48:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:48:15.678+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:48:15.984 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:48:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:48:15.679+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:48:16.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:48:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:48:15.982+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:48:16.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:48:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:48:15.982+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:48:16.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:48:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:48:15.982+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:48:16.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:48:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:48:15.982+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:48:16.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:48:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:48:16.093+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:48:16.321 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:48:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:48:16.093+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:48:16.321 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:48:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:48:16.093+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:48:16.321 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:48:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:48:16.094+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:48:16.983 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:48:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:48:16.698+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:48:16.983 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:48:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:48:16.698+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:48:16.983 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:48:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:48:16.698+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:48:16.983 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:48:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:48:16.699+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:48:17.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:48:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:48:17.091+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:48:17.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:48:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:48:17.091+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:48:17.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:48:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:48:17.091+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:48:17.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:48:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:48:17.091+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:48:17.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:48:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:48:16.981+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:48:17.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:48:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:48:16.981+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:48:17.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:48:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:48:16.981+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:48:17.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:48:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:48:16.982+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:48:18.046 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:48:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:48:17.730+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:48:18.046 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:48:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:48:17.730+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:48:18.046 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:48:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:48:17.730+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:48:18.047 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:48:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:48:17.731+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:48:18.047 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:48:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:48:17.971+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:48:18.047 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:48:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:48:17.972+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:48:18.047 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:48:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:48:17.972+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:48:18.047 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:48:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:48:17.972+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:48:18.319 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:48:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:48:18.045+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:48:18.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:48:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:48:18.045+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:48:18.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:48:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:48:18.045+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:48:18.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:48:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:48:18.045+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:48:18.991 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:48:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:48:18.697+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:48:18.991 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:48:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:48:18.697+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:48:18.991 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:48:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:48:18.697+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:48:18.991 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:48:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:48:18.698+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:48:19.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:48:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:48:18.989+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:48:19.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:48:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:48:18.989+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:48:19.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:48:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:48:18.989+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:48:19.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:48:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:48:18.990+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:48:19.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:48:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:48:19.032+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:48:19.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:48:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:48:19.032+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:48:19.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:48:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:48:19.032+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:48:19.321 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:48:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:48:19.033+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:48:20.019 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:48:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:48:19.670+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:48:20.020 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:48:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:48:19.670+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:48:20.020 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:48:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:48:19.670+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:48:20.020 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:48:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:48:19.671+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:48:20.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:48:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:48:20.071+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:48:20.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:48:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:48:20.071+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:48:20.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:48:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:48:20.071+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:48:20.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:48:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:48:20.071+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:48:20.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:48:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:48:20.018+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:48:20.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:48:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:48:20.018+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:48:20.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:48:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:48:20.018+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:48:20.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:48:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:48:20.018+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:48:21.037 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:48:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:48:20.591+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 25500 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:48:21.038 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:48:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:48:20.716+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:48:21.038 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:48:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:48:20.716+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:48:21.038 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:48:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:48:20.716+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:48:21.038 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:48:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:48:20.717+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:48:21.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:48:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:48:21.052+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:48:21.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:48:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:48:21.052+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:48:21.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:48:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:48:21.052+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:48:21.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:48:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:48:21.052+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:48:21.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:48:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:48:21.036+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:48:21.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:48:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:48:21.036+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:48:21.321 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:48:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:48:21.036+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:48:21.321 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:48:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:48:21.037+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:48:22.023 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:48:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:48:21.706+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:48:22.024 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:48:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:48:21.706+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:48:22.024 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:48:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:48:21.706+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:48:22.024 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:48:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:48:21.707+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:48:22.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:48:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:48:22.050+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:48:22.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:48:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:48:22.050+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:48:22.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:48:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:48:22.050+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:48:22.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:48:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:48:22.050+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:48:22.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:48:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:48:22.022+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:48:22.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:48:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:48:22.022+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:48:22.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:48:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:48:22.022+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:48:22.321 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:48:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:48:22.023+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:48:23.059 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:48:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:48:22.671+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:48:23.059 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:48:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:48:22.671+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:48:23.059 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:48:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:48:22.671+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:48:23.059 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:48:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:48:22.672+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:48:23.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:48:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:48:23.071+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:48:23.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:48:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:48:23.071+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:48:23.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:48:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:48:23.071+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:48:23.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:48:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:48:23.071+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:48:23.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:48:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:48:23.058+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:48:23.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:48:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:48:23.058+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:48:23.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:48:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:48:23.058+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:48:23.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:48:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:48:23.058+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:48:24.033 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:48:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:48:23.645+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:48:24.034 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:48:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:48:23.645+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:48:24.034 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:48:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:48:23.645+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:48:24.035 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:48:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:48:23.646+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:48:24.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:48:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:48:24.048+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:48:24.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:48:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:48:24.048+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:48:24.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:48:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:48:24.048+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:48:24.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:48:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:48:24.048+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:48:24.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:48:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:48:24.032+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:48:24.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:48:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:48:24.032+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:48:24.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:48:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:48:24.032+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:48:24.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:48:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:48:24.032+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:48:25.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:48:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:48:24.674+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:48:25.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:48:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:48:24.674+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:48:25.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:48:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:48:24.674+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:48:25.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:48:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:48:24.675+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:48:25.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:48:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:48:25.067+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:48:25.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:48:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:48:25.067+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:48:25.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:48:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:48:25.067+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:48:25.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:48:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:48:25.068+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:48:25.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:48:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:48:25.078+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:48:25.321 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:48:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:48:25.078+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:48:25.321 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:48:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:48:25.078+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:48:25.321 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:48:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:48:25.079+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:48:26.057 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:48:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:48:25.594+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 25584 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:48:26.057 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:48:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:48:25.658+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:48:26.057 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:48:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:48:25.658+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:48:26.057 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:48:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:48:25.658+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:48:26.057 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:48:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:48:25.659+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:48:26.101 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:48:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:48:26.055+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:48:26.102 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:48:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:48:26.055+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:48:26.102 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:48:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:48:26.055+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:48:26.102 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:48:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:48:26.056+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:48:26.102 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:48:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:48:26.099+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:48:26.102 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:48:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:48:26.099+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:48:26.102 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:48:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:48:26.099+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:48:26.102 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:48:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:48:26.099+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:48:27.035 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:48:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:48:26.671+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:48:27.035 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:48:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:48:26.671+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:48:27.035 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:48:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:48:26.671+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:48:27.036 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:48:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:48:26.671+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:48:27.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:48:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:48:27.033+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:48:27.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:48:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:48:27.034+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:48:27.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:48:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:48:27.034+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:48:27.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:48:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:48:27.034+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:48:27.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:48:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:48:27.107+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:48:27.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:48:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:48:27.107+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:48:27.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:48:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:48:27.107+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:48:27.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:48:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:48:27.108+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:48:28.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:48:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:48:27.697+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:48:28.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:48:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:48:27.697+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:48:28.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:48:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:48:27.697+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:48:28.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:48:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:48:27.697+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:48:28.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:48:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:48:28.074+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:48:28.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:48:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:48:28.074+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:48:28.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:48:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:48:28.074+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:48:28.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:48:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:48:28.075+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:48:28.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:48:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:48:28.102+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:48:28.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:48:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:48:28.102+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:48:28.571 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:48:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:48:28.102+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:48:28.571 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:48:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:48:28.103+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:48:29.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:48:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:48:28.716+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:48:29.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:48:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:48:28.716+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:48:29.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:48:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:48:28.716+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:48:29.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:48:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:48:28.717+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:48:29.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:48:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:48:29.087+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:48:29.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:48:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:48:29.087+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:48:29.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:48:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:48:29.087+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:48:29.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:48:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:48:29.088+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:48:29.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:48:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:48:29.135+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:48:29.571 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:48:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:48:29.135+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:48:29.571 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:48:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:48:29.135+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:48:29.571 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:48:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:48:29.136+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:48:30.066 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:48:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:48:29.708+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:48:30.066 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:48:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:48:29.708+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:48:30.066 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:48:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:48:29.708+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:48:30.067 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:48:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:48:29.709+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:48:30.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:48:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:48:30.149+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:48:30.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:48:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:48:30.149+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:48:30.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:48:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:48:30.149+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:48:30.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:48:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:48:30.150+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:48:30.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:48:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:48:30.064+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:48:30.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:48:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:48:30.064+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:48:30.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:48:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:48:30.064+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:48:30.321 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:48:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:48:30.064+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:48:31.027 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:48:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:48:30.595+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 25666 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:48:31.027 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:48:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:48:30.675+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:48:31.027 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:48:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:48:30.675+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:48:31.028 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:48:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:48:30.675+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:48:31.028 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:48:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:48:30.676+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:48:31.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:48:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:48:31.025+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:48:31.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:48:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:48:31.025+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:48:31.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:48:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:48:31.025+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:48:31.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:48:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:48:31.025+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:48:31.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:48:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:48:31.170+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:48:31.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:48:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:48:31.170+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:48:31.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:48:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:48:31.170+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:48:31.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:48:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:48:31.171+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:48:32.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:48:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:48:31.661+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:48:32.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:48:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:48:31.661+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:48:32.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:48:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:48:31.661+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:48:32.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:48:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:48:31.662+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:48:32.411 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:48:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:48:32.068+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:48:32.412 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:48:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:48:32.068+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:48:32.412 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:48:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:48:32.068+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:48:32.412 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:48:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:48:32.068+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:48:32.412 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:48:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:48:32.186+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:48:32.412 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:48:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:48:32.186+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:48:32.413 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:48:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:48:32.186+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:48:32.413 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:48:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:48:32.187+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:48:33.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:48:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:48:32.690+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:48:33.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:48:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:48:32.690+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:48:33.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:48:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:48:32.690+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:48:33.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:48:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:48:32.691+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:48:33.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:48:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:48:33.089+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:48:33.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:48:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:48:33.089+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:48:33.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:48:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:48:33.089+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:48:33.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:48:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:48:33.090+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:48:33.571 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:48:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:48:33.233+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:48:33.571 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:48:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:48:33.233+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:48:33.571 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:48:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:48:33.233+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:48:33.571 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:48:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:48:33.233+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:48:34.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:48:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:48:33.644+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:48:34.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:48:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:48:33.644+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:48:34.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:48:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:48:33.644+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:48:34.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:48:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:48:33.644+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:48:34.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:48:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:48:34.075+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:48:34.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:48:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:48:34.075+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:48:34.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:48:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:48:34.075+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:48:34.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:48:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:48:34.076+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:48:34.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:48:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:48:34.249+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:48:34.571 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:48:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:48:34.249+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:48:34.571 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:48:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:48:34.249+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:48:34.571 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:48:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:48:34.250+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:48:35.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:48:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:48:34.648+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:48:35.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:48:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:48:34.648+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:48:35.071 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:48:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:48:34.648+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:48:35.071 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:48:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:48:34.649+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:48:35.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:48:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:48:35.116+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:48:35.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:48:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:48:35.116+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:48:35.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:48:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:48:35.116+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:48:35.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:48:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:48:35.117+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:48:35.571 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:48:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:48:35.290+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:48:35.571 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:48:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:48:35.290+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:48:35.571 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:48:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:48:35.290+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:48:35.571 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:48:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:48:35.290+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:48:36.070 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:48:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:48:35.598+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 25750 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:48:36.074 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:48:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:48:35.681+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:48:36.075 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:48:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:48:35.681+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:48:36.075 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:48:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:48:35.681+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:48:36.075 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:48:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:48:35.682+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:48:36.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:48:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:48:36.300+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:48:36.571 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:48:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:48:36.300+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:48:36.572 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:48:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:48:36.300+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:48:36.572 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:48:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:48:36.300+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:48:36.573 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:48:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:48:36.143+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:48:36.573 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:48:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:48:36.143+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:48:36.576 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:48:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:48:36.143+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:48:36.576 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:48:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:48:36.143+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:48:37.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:48:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:48:36.645+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:48:37.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:48:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:48:36.645+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:48:37.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:48:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:48:36.645+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:48:37.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:48:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:48:36.645+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:48:37.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:48:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:48:37.315+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:48:37.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:48:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:48:37.315+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:48:37.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:48:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:48:37.315+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:48:37.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:48:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:48:37.315+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:48:37.571 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:48:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:48:37.175+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:48:37.571 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:48:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:48:37.175+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:48:37.571 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:48:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:48:37.175+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:48:37.571 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:48:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:48:37.175+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:48:38.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:48:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:48:37.651+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:48:38.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:48:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:48:37.651+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:48:38.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:48:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:48:37.651+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:48:38.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:48:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:48:37.652+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:48:38.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:48:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:48:38.223+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:48:38.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:48:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:48:38.223+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:48:38.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:48:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:48:38.223+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:48:38.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:48:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:48:38.224+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:48:38.571 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:48:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:48:38.340+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:48:38.571 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:48:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:48:38.340+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:48:38.571 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:48:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:48:38.340+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:48:38.571 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:48:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:48:38.341+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:48:39.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:48:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:48:38.622+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:48:39.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:48:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:48:38.622+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:48:39.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:48:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:48:38.622+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:48:39.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:48:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:48:38.623+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:48:39.368 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:48:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:48:39.244+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:48:39.368 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:48:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:48:39.244+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:48:39.368 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:48:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:48:39.244+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:48:39.369 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:48:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:48:39.245+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:48:39.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:48:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:48:39.582+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:48:39.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:48:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:48:39.582+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:48:39.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:48:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:48:39.582+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:48:39.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:48:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:48:39.583+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:48:39.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:48:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:48:39.366+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:48:39.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:48:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:48:39.366+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:48:39.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:48:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:48:39.366+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:48:39.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:48:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:48:39.367+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:48:40.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:48:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:48:40.226+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:48:40.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:48:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:48:40.226+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:48:40.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:48:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:48:40.226+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:48:40.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:48:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:48:40.227+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:48:40.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:48:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:48:40.416+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:48:40.571 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:48:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:48:40.416+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:48:40.571 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:48:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:48:40.416+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:48:40.571 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:48:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:48:40.417+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:48:41.070 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:48:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:48:40.602+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 25832 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:48:41.071 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:48:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:48:40.577+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:48:41.071 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:48:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:48:40.577+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:48:41.071 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:48:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:48:40.577+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:48:41.071 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:48:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:48:40.577+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:48:41.565 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:48:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:48:41.267+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:48:41.566 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:48:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:48:41.267+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:48:41.566 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:48:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:48:41.267+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:48:41.566 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:48:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:48:41.267+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:48:41.566 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:48:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:48:41.381+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:48:41.567 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:48:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:48:41.381+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:48:41.567 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:48:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:48:41.381+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:48:41.567 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:48:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:48:41.381+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:48:41.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:48:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:48:41.564+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:48:41.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:48:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:48:41.564+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:48:41.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:48:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:48:41.564+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:48:41.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:48:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:48:41.565+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:48:42.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:48:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:48:42.248+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:48:42.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:48:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:48:42.248+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:48:42.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:48:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:48:42.248+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:48:42.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:48:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:48:42.248+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:48:42.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:48:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:48:42.393+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:48:42.571 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:48:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:48:42.393+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:48:42.571 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:48:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:48:42.393+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:48:42.571 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:48:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:48:42.393+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:48:43.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:48:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:48:42.573+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:48:43.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:48:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:48:42.573+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:48:43.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:48:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:48:42.574+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:48:43.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:48:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:48:42.575+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:48:43.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:48:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:48:43.344+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:48:43.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:48:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:48:43.344+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:48:43.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:48:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:48:43.344+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:48:43.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:48:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:48:43.345+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:48:43.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:48:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:48:43.221+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:48:43.571 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:48:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:48:43.221+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:48:43.571 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:48:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:48:43.221+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:48:43.571 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:48:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:48:43.222+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:48:44.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:48:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:48:43.569+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:48:44.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:48:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:48:43.569+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:48:44.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:48:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:48:43.569+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:48:44.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:48:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:48:43.569+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:48:44.373 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:48:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:48:44.197+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:48:44.373 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:48:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:48:44.197+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:48:44.373 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:48:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:48:44.197+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:48:44.374 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:48:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:48:44.197+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:48:44.699 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:48:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:48:44.372+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:48:44.699 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:48:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:48:44.372+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:48:44.700 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:48:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:48:44.372+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:48:44.700 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:48:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:48:44.372+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:48:44.700 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:48:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:48:44.595+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:48:44.700 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:48:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:48:44.595+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:48:44.700 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:48:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:48:44.595+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:48:44.700 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:48:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:48:44.596+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:48:45.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:48:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:48:45.241+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:48:45.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:48:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:48:45.241+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:48:45.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:48:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:48:45.241+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:48:45.571 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:48:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:48:45.242+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:48:45.571 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:48:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:48:45.407+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:48:45.571 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:48:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:48:45.407+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:48:45.571 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:48:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:48:45.407+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:48:45.571 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:48:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:48:45.407+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:48:46.070 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:48:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:48:45.605+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 25916 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:48:46.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:48:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:48:45.608+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:48:46.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:48:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:48:45.608+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:48:46.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:48:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:48:45.608+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:48:46.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:48:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:48:45.609+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:48:46.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:48:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:48:46.416+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:48:46.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:48:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:48:46.416+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:48:46.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:48:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:48:46.416+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:48:46.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:48:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:48:46.417+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:48:46.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:48:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:48:46.266+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:48:46.571 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:48:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:48:46.266+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:48:46.571 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:48:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:48:46.266+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:48:46.571 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:48:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:48:46.267+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:48:47.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:48:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:48:46.644+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:48:47.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:48:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:48:46.644+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:48:47.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:48:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:48:46.644+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:48:47.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:48:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:48:46.645+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:48:47.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:48:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:48:47.401+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:48:47.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:48:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:48:47.401+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:48:47.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:48:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:48:47.401+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:48:47.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:48:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:48:47.401+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:48:47.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:48:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:48:47.233+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:48:47.571 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:48:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:48:47.233+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:48:47.571 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:48:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:48:47.233+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:48:47.571 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:48:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:48:47.233+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:48:48.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:48:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:48:47.641+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:48:48.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:48:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:48:47.641+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:48:48.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:48:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:48:47.641+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:48:48.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:48:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:48:47.641+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:48:48.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:48:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:48:48.395+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:48:48.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:48:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:48:48.395+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:48:48.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:48:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:48:48.395+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:48:48.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:48:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:48:48.395+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:48:48.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:48:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:48:48.269+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:48:48.571 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:48:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:48:48.269+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:48:48.571 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:48:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:48:48.269+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:48:48.571 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:48:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:48:48.269+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:48:49.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:48:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:48:48.689+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:48:49.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:48:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:48:48.689+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:48:49.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:48:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:48:48.689+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:48:49.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:48:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:48:48.690+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:48:49.366 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:48:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:48:49.314+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:48:49.367 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:48:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:48:49.314+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:48:49.367 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:48:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:48:49.314+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:48:49.367 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:48:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:48:49.315+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:48:49.710 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:48:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:48:49.365+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:48:49.710 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:48:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:48:49.365+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:48:49.710 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:48:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:48:49.365+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:48:49.711 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:48:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:48:49.366+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:48:50.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:48:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:48:49.709+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:48:50.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:48:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:48:49.709+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:48:50.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:48:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:48:49.709+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:48:50.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:48:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:48:49.710+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:48:50.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:48:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:48:50.309+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:48:50.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:48:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:48:50.309+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:48:50.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:48:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:48:50.309+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:48:50.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:48:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:48:50.309+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:48:50.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:48:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:48:50.413+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:48:50.571 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:48:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:48:50.413+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:48:50.571 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:48:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:48:50.413+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:48:50.571 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:48:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:48:50.413+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:48:51.070 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:48:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:48:50.607+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 25998 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:48:51.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:48:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:48:50.724+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:48:51.071 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:48:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:48:50.724+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:48:51.071 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:48:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:48:50.724+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:48:51.071 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:48:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:48:50.725+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:48:51.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:48:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:48:51.274+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:48:51.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:48:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:48:51.274+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:48:51.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:48:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:48:51.274+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:48:51.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:48:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:48:51.274+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:48:51.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:48:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:48:51.444+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:48:51.571 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:48:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:48:51.444+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:48:51.571 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:48:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:48:51.444+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:48:51.571 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:48:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:48:51.445+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:48:52.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:48:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:48:51.748+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:48:52.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:48:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:48:51.748+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:48:52.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:48:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:48:51.748+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:48:52.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:48:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:48:51.748+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:48:52.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:48:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:48:52.282+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:48:52.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:48:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:48:52.282+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:48:52.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:48:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:48:52.282+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:48:52.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:48:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:48:52.282+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:48:52.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:48:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:48:52.429+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:48:52.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:48:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:48:52.429+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:48:52.571 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:48:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:48:52.429+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:48:52.571 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:48:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:48:52.429+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:48:53.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:48:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:48:52.753+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:48:53.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:48:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:48:52.753+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:48:53.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:48:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:48:52.753+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:48:53.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:48:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:48:52.754+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:48:53.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:48:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:48:53.260+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:48:53.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:48:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:48:53.260+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:48:53.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:48:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:48:53.260+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:48:53.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:48:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:48:53.261+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:48:53.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:48:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:48:53.445+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:48:53.571 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:48:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:48:53.445+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:48:53.571 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:48:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:48:53.445+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:48:53.571 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:48:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:48:53.446+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:48:54.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:48:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:48:53.767+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:48:54.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:48:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:48:53.767+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:48:54.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:48:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:48:53.767+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:48:54.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:48:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:48:53.768+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:48:54.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:48:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:48:54.444+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:48:54.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:48:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:48:54.444+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:48:54.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:48:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:48:54.444+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:48:54.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:48:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:48:54.444+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:48:54.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:48:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:48:54.256+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:48:54.571 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:48:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:48:54.256+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:48:54.571 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:48:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:48:54.256+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:48:54.571 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:48:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:48:54.257+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:48:55.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:48:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:48:54.810+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:48:55.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:48:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:48:54.810+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:48:55.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:48:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:48:54.810+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:48:55.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:48:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:48:54.811+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:48:55.489 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:48:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:48:55.208+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:48:55.489 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:48:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:48:55.208+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:48:55.489 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:48:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:48:55.208+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:48:55.489 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:48:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:48:55.209+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:48:55.771 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:48:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:48:55.610+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 26082 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:48:55.771 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:48:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:48:55.487+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:48:55.771 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:48:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:48:55.487+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:48:55.772 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:48:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:48:55.487+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:48:55.772 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:48:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:48:55.488+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:48:56.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:48:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:48:55.769+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:48:56.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:48:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:48:55.769+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:48:56.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:48:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:48:55.769+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:48:56.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:48:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:48:55.770+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:48:56.520 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:48:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:48:56.180+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:48:56.521 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:48:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:48:56.180+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:48:56.521 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:48:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:48:56.180+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:48:56.521 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:48:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:48:56.181+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:48:56.776 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:48:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:48:56.518+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:48:56.777 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:48:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:48:56.518+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:48:56.777 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:48:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:48:56.518+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:48:56.777 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:48:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:48:56.519+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:48:57.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:48:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:48:56.775+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:48:57.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:48:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:48:56.775+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:48:57.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:48:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:48:56.775+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:48:57.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:48:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:48:56.776+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:48:57.562 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:48:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:48:57.224+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:48:57.562 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:48:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:48:57.224+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:48:57.562 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:48:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:48:57.224+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:48:57.563 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:48:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:48:57.225+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:48:57.821 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:48:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:48:57.767+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:48:57.821 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:48:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:48:57.767+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:48:57.821 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:48:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:48:57.767+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:48:57.821 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:48:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:48:57.767+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:48:57.822 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:48:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:48:57.561+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:48:57.822 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:48:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:48:57.561+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:48:57.822 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:48:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:48:57.561+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:48:57.822 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:48:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:48:57.561+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:48:58.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:48:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:48:58.269+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:48:58.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:48:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:48:58.269+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:48:58.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:48:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:48:58.269+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:48:58.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:48:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:48:58.270+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:48:59.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:48:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:48:58.757+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:48:59.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:48:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:48:58.757+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:48:59.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:48:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:48:58.757+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:48:59.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:48:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:48:58.758+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:48:59.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:48:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:48:58.585+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:48:59.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:48:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:48:58.585+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:48:59.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:48:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:48:58.585+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:48:59.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:48:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:48:58.585+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:48:59.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:48:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:48:59.230+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:48:59.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:48:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:48:59.230+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:48:59.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:48:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:48:59.230+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:48:59.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:48:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:48:59.231+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:49:00.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:48:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:48:59.734+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:49:00.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:48:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:48:59.734+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:49:00.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:48:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:48:59.734+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:49:00.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:48:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:48:59.735+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:49:00.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:48:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:48:59.591+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:49:00.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:48:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:48:59.591+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:49:00.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:48:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:48:59.591+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:49:00.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:48:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:48:59.591+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:49:00.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:49:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:49:00.204+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:49:00.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:49:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:49:00.204+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:49:00.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:49:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:49:00.204+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:49:00.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:49:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:49:00.205+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:49:01.070 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:49:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:49:00.613+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 26164 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:49:01.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:49:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:49:00.640+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:49:01.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:49:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:49:00.640+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:49:01.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:49:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:49:00.640+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:49:01.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:49:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:49:00.641+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:49:01.071 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:49:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:49:00.722+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:49:01.071 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:49:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:49:00.722+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:49:01.071 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:49:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:49:00.722+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:49:01.071 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:49:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:49:00.723+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:49:01.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:49:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:49:01.155+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:49:01.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:49:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:49:01.155+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:49:01.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:49:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:49:01.155+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:49:01.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:49:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:49:01.156+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:49:02.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:49:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:49:01.742+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:49:02.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:49:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:49:01.742+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:49:02.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:49:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:49:01.742+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:49:02.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:49:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:49:01.743+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:49:02.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:49:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:49:01.671+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:49:02.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:49:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:49:01.671+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:49:02.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:49:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:49:01.671+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:49:02.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:49:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:49:01.672+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:49:02.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:49:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:49:02.195+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:49:02.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:49:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:49:02.195+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:49:02.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:49:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:49:02.195+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:49:02.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:49:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:49:02.196+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:49:03.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:49:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:49:02.767+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:49:03.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:49:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:49:02.767+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:49:03.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:49:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:49:02.767+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:49:03.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:49:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:49:02.768+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:49:03.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:49:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:49:02.721+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:49:03.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:49:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:49:02.721+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:49:03.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:49:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:49:02.721+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:49:03.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:49:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:49:02.722+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:49:03.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:49:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:49:03.201+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:49:03.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:49:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:49:03.201+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:49:03.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:49:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:49:03.201+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:49:03.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:49:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:49:03.202+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:49:04.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:49:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:49:03.814+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:49:04.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:49:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:49:03.815+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:49:04.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:49:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:49:03.815+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:49:04.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:49:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:49:03.815+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:49:04.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:49:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:49:03.714+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:49:04.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:49:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:49:03.714+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:49:04.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:49:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:49:03.714+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:49:04.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:49:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:49:03.715+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:49:04.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:49:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:49:04.208+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:49:04.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:49:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:49:04.208+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:49:04.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:49:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:49:04.208+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:49:04.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:49:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:49:04.208+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:49:05.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:49:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:49:04.838+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:49:05.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:49:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:49:04.838+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:49:05.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:49:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:49:04.838+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:49:05.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:49:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:49:04.839+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:49:05.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:49:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:49:04.710+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:49:05.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:49:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:49:04.710+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:49:05.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:49:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:49:04.710+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:49:05.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:49:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:49:04.710+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:49:05.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:49:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:49:05.189+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:49:05.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:49:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:49:05.189+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:49:05.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:49:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:49:05.189+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:49:05.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:49:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:49:05.190+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:49:06.069 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:49:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:49:05.616+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 26248 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:49:06.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:49:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:49:05.664+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:49:06.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:49:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:49:05.664+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:49:06.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:49:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:49:05.664+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:49:06.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:49:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:49:05.665+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:49:06.071 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:49:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:49:05.832+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:49:06.071 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:49:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:49:05.832+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:49:06.071 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:49:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:49:05.832+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:49:06.071 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:49:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:49:05.833+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:49:06.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:49:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:49:06.172+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:49:06.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:49:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:49:06.172+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:49:06.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:49:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:49:06.172+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:49:06.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:49:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:49:06.173+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:49:07.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:49:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:49:06.818+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:49:07.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:49:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:49:06.818+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:49:07.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:49:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:49:06.818+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:49:07.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:49:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:49:06.819+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:49:07.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:49:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:49:06.701+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:49:07.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:49:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:49:06.701+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:49:07.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:49:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:49:06.701+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:49:07.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:49:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:49:06.701+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:49:07.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:49:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:49:07.174+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:49:07.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:49:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:49:07.174+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:49:07.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:49:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:49:07.174+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:49:07.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:49:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:49:07.175+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:49:08.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:49:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:49:07.805+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:49:08.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:49:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:49:07.805+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:49:08.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:49:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:49:07.805+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:49:08.071 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:49:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:49:07.806+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:49:08.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:49:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:49:07.677+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:49:08.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:49:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:49:07.677+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:49:08.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:49:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:49:07.677+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:49:08.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:49:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:49:07.677+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:49:08.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:49:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:49:08.144+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:49:08.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:49:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:49:08.144+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:49:08.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:49:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:49:08.144+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:49:08.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:49:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:49:08.144+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:49:09.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:49:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:49:08.832+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:49:09.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:49:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:49:08.832+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:49:09.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:49:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:49:08.832+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:49:09.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:49:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:49:08.834+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:49:09.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:49:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:49:08.651+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:49:09.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:49:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:49:08.651+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:49:09.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:49:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:49:08.651+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:49:09.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:49:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:49:08.652+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:49:09.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:49:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:49:09.113+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:49:09.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:49:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:49:09.113+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:49:09.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:49:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:49:09.113+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:49:09.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:49:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:49:09.114+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:49:10.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:49:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:49:09.836+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:49:10.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:49:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:49:09.836+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:49:10.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:49:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:49:09.836+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:49:10.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:49:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:49:09.837+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:49:10.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:49:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:49:09.677+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:49:10.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:49:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:49:09.677+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:49:10.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:49:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:49:09.677+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:49:10.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:49:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:49:09.678+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:49:10.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:49:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:49:10.077+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:49:10.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:49:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:49:10.077+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:49:10.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:49:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:49:10.077+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:49:10.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:49:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:49:10.078+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:49:11.069 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:49:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:49:10.619+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 26330 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:49:11.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:49:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:49:10.678+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:49:11.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:49:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:49:10.678+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:49:11.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:49:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:49:10.678+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:49:11.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:49:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:49:10.678+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:49:11.071 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:49:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:49:10.867+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:49:11.071 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:49:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:49:10.867+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:49:11.071 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:49:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:49:10.867+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:49:11.071 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:49:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:49:10.868+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:49:11.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:49:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:49:11.123+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:49:11.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:49:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:49:11.123+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:49:11.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:49:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:49:11.123+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:49:11.571 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:49:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:49:11.124+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:49:12.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:49:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:49:11.669+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:49:12.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:49:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:49:11.669+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:49:12.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:49:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:49:11.669+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:49:12.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:49:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:49:11.669+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:49:12.071 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:49:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:49:11.911+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:49:12.071 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:49:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:49:11.911+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:49:12.071 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:49:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:49:11.911+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:49:12.071 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:49:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:49:11.911+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:49:12.347 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:49:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:49:12.080+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:49:12.347 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:49:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:49:12.080+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:49:12.348 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:49:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:49:12.080+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:49:12.348 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:49:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:49:12.081+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:49:13.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:49:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:49:12.905+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:49:13.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:49:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:49:12.905+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:49:13.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:49:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:49:12.905+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:49:13.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:49:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:49:12.906+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:49:13.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:49:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:49:12.695+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:49:13.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:49:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:49:12.695+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:49:13.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:49:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:49:12.695+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:49:13.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:49:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:49:12.695+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:49:13.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:49:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:49:13.127+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:49:13.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:49:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:49:13.127+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:49:13.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:49:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:49:13.127+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:49:13.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:49:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:49:13.128+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:49:14.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:49:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:49:13.907+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:49:14.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:49:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:49:13.907+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:49:14.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:49:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:49:13.907+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:49:14.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:49:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:49:13.908+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:49:14.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:49:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:49:13.684+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:49:14.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:49:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:49:13.684+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:49:14.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:49:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:49:13.684+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:49:14.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:49:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:49:13.684+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:49:14.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:49:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:49:14.115+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:49:14.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:49:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:49:14.115+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:49:14.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:49:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:49:14.115+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:49:14.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:49:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:49:14.116+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:49:15.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:49:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:49:14.952+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:49:15.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:49:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:49:14.952+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:49:15.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:49:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:49:14.952+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:49:15.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:49:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:49:14.953+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:49:15.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:49:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:49:14.726+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:49:15.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:49:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:49:14.726+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:49:15.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:49:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:49:14.726+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:49:15.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:49:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:49:14.727+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:49:15.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:49:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:49:15.150+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:49:15.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:49:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:49:15.150+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:49:15.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:49:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:49:15.150+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:49:15.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:49:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:49:15.151+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:49:15.967 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:49:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:49:15.622+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 26414 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:49:15.967 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:49:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:49:15.716+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:49:15.967 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:49:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:49:15.716+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:49:15.967 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:49:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:49:15.716+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:49:15.967 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:49:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:49:15.717+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:49:16.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:49:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:49:16.154+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:49:16.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:49:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:49:16.154+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:49:16.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:49:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:49:16.154+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:49:16.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:49:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:49:16.155+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:49:16.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:49:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:49:15.965+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:49:16.321 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:49:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:49:15.965+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:49:16.321 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:49:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:49:15.965+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:49:16.321 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:49:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:49:15.966+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:49:17.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:49:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:49:16.989+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:49:17.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:49:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:49:16.989+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:49:17.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:49:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:49:16.989+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:49:17.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:49:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:49:16.990+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:49:17.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:49:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:49:16.751+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:49:17.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:49:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:49:16.751+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:49:17.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:49:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:49:16.751+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:49:17.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:49:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:49:16.752+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:49:17.571 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:49:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:49:17.203+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:49:17.571 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:49:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:49:17.203+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:49:17.571 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:49:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:49:17.203+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:49:17.571 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:49:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:49:17.203+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:49:18.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:49:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:49:17.986+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:49:18.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:49:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:49:17.986+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:49:18.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:49:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:49:17.986+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:49:18.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:49:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:49:17.987+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:49:18.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:49:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:49:17.797+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:49:18.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:49:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:49:17.797+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:49:18.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:49:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:49:17.797+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:49:18.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:49:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:49:17.798+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:49:18.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:49:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:49:18.198+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:49:18.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:49:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:49:18.198+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:49:18.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:49:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:49:18.198+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:49:18.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:49:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:49:18.199+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:49:19.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:49:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:49:18.991+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:49:19.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:49:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:49:18.991+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:49:19.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:49:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:49:18.991+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:49:19.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:49:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:49:18.992+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:49:19.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:49:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:49:18.814+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:49:19.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:49:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:49:18.814+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:49:19.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:49:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:49:18.814+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:49:19.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:49:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:49:18.815+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:49:19.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:49:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:49:19.203+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:49:19.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:49:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:49:19.203+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:49:19.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:49:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:49:19.203+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:49:19.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:49:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:49:19.204+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:49:20.204 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:49:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:49:20.009+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:49:20.204 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:49:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:49:20.009+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:49:20.204 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:49:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:49:20.009+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:49:20.205 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:49:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:49:20.010+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:49:20.205 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:49:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:49:19.836+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:49:20.205 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:49:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:49:19.836+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:49:20.205 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:49:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:49:19.836+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:49:20.205 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:49:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:49:19.836+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:49:20.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:49:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:49:20.242+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:49:20.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:49:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:49:20.242+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:49:20.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:49:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:49:20.242+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:49:20.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:49:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:49:20.243+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:49:21.021 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:49:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:49:20.624+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 26496 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:49:21.021 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:49:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:49:20.846+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:49:21.022 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:49:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:49:20.846+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:49:21.022 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:49:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:49:20.846+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:49:21.022 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:49:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:49:20.847+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:49:21.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:49:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:49:21.020+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:49:21.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:49:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:49:21.020+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:49:21.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:49:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:49:21.020+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:49:21.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:49:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:49:21.021+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:49:21.321 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:49:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:49:21.228+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:49:21.321 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:49:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:49:21.228+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:49:21.321 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:49:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:49:21.228+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:49:21.321 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:49:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:49:21.228+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:49:22.265 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:49:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:49:22.063+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:49:22.266 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:49:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:49:22.063+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:49:22.266 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:49:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:49:22.063+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:49:22.266 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:49:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:49:22.064+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:49:22.266 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:49:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:49:21.890+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:49:22.266 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:49:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:49:21.890+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:49:22.266 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:49:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:49:21.890+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:49:22.266 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:49:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:49:21.891+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:49:22.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:49:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:49:22.264+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:49:22.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:49:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:49:22.264+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:49:22.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:49:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:49:22.264+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:49:22.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:49:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:49:22.265+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:49:23.229 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:49:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:49:23.071+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:49:23.229 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:49:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:49:23.071+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:49:23.230 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:49:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:49:23.071+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:49:23.230 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:49:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:49:23.072+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:49:23.230 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:49:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:49:22.854+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:49:23.230 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:49:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:49:22.854+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:49:23.230 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:49:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:49:22.854+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:49:23.230 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:49:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:49:22.854+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:49:23.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:49:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:49:23.228+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:49:23.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:49:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:49:23.228+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:49:23.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:49:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:49:23.228+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:49:23.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:49:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:49:23.228+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:49:24.113 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:49:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:49:23.840+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:49:24.114 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:49:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:49:23.840+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:49:24.114 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:49:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:49:23.840+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:49:24.114 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:49:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:49:23.841+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:49:24.570 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:49:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:49:24.112+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:49:24.570 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:49:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:49:24.112+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:49:24.570 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:49:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:49:24.112+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:49:24.570 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:49:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:49:24.113+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:49:24.571 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:49:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:49:24.216+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:49:24.571 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:49:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:49:24.216+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:49:24.571 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:49:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:49:24.216+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:49:24.571 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:49:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:49:24.216+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:49:25.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:49:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:49:24.828+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:49:25.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:49:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:49:24.828+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:49:25.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:49:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:49:24.828+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:49:25.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:49:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:49:24.829+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:49:25.570 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:49:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:49:25.136+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:49:25.570 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:49:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:49:25.136+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:49:25.570 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:49:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:49:25.136+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:49:25.570 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:49:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:49:25.137+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:49:25.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:49:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:49:25.193+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:49:25.571 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:49:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:49:25.193+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:49:25.571 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:49:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:49:25.193+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:49:25.571 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:49:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:49:25.194+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:49:26.069 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:49:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:49:25.627+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 26580 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:49:26.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:49:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:49:25.832+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:49:26.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:49:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:49:25.832+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:49:26.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:49:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:49:25.832+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:49:26.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:49:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:49:25.833+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:49:26.569 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:49:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:49:26.158+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:49:26.570 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:49:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:49:26.158+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:49:26.570 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:49:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:49:26.158+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:49:26.570 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:49:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:49:26.159+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:49:26.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:49:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:49:26.236+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:49:26.571 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:49:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:49:26.236+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:49:26.571 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:49:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:49:26.236+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:49:26.571 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:49:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:49:26.237+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:49:27.163 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:49:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:49:26.858+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:49:27.163 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:49:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:49:26.858+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:49:27.163 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:49:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:49:26.859+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:49:27.163 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:49:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:49:26.859+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:49:27.570 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:49:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:49:27.160+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:49:27.570 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:49:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:49:27.160+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:49:27.570 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:49:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:49:27.160+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:49:27.570 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:49:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:49:27.162+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:49:27.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:49:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:49:27.237+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:49:27.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:49:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:49:27.237+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:49:27.571 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:49:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:49:27.237+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:49:27.571 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:49:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:49:27.238+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:49:28.182 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:49:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:49:27.864+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:49:28.183 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:49:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:49:27.864+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:49:28.183 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:49:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:49:27.864+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:49:28.183 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:49:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:49:27.864+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:49:28.570 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:49:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:49:28.180+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:49:28.570 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:49:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:49:28.180+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:49:28.570 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:49:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:49:28.180+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:49:28.570 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:49:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:49:28.181+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:49:28.571 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:49:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:49:28.205+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:49:28.571 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:49:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:49:28.205+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:49:28.571 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:49:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:49:28.205+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:49:28.571 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:49:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:49:28.205+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:49:29.168 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:49:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:49:28.909+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:49:29.169 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:49:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:49:28.909+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:49:29.169 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:49:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:49:28.909+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:49:29.169 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:49:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:49:28.909+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:49:29.570 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:49:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:49:29.199+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:49:29.570 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:49:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:49:29.199+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:49:29.570 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:49:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:49:29.199+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:49:29.570 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:49:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:49:29.200+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:49:29.571 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:49:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:49:29.166+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:49:29.571 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:49:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:49:29.166+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:49:29.571 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:49:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:49:29.166+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:49:29.571 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:49:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:49:29.167+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:49:30.193 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:49:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:49:29.884+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:49:30.194 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:49:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:49:29.884+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:49:30.194 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:49:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:49:29.884+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:49:30.194 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:49:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:49:29.885+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:49:30.570 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:49:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:49:30.191+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:49:30.570 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:49:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:49:30.191+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:49:30.570 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:49:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:49:30.191+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:49:30.570 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:49:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:49:30.192+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:49:30.571 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:49:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:49:30.211+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:49:30.571 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:49:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:49:30.211+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:49:30.571 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:49:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:49:30.211+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:49:30.571 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:49:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:49:30.212+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:49:30.883 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:49:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:49:30.629+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 26661 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:49:31.153 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:49:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:49:30.881+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:49:31.153 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:49:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:49:30.881+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:49:31.153 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:49:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:49:30.881+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:49:31.154 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:49:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:49:30.882+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:49:31.570 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:49:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:49:31.151+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:49:31.570 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:49:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:49:31.151+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:49:31.570 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:49:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:49:31.151+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:49:31.570 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:49:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:49:31.152+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:49:31.571 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:49:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:49:31.255+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:49:31.571 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:49:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:49:31.255+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:49:31.571 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:49:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:49:31.255+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:49:31.571 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:49:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:49:31.256+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:49:32.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:49:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:49:31.854+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:49:32.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:49:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:49:31.854+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:49:32.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:49:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:49:31.854+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:49:32.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:49:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:49:31.854+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:49:32.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:49:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:49:32.241+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:49:32.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:49:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:49:32.241+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:49:32.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:49:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:49:32.241+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:49:32.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:49:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:49:32.242+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:49:32.571 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:49:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:49:32.132+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:49:32.571 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:49:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:49:32.132+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:49:32.571 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:49:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:49:32.132+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:49:32.571 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:49:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:49:32.133+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:49:33.167 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:49:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:49:32.879+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:49:33.167 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:49:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:49:32.879+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:49:33.167 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:49:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:49:32.879+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:49:33.167 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:49:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:49:32.879+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:49:33.570 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:49:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:49:33.163+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:49:33.570 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:49:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:49:33.163+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:49:33.570 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:49:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:49:33.163+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:49:33.571 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:49:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:49:33.164+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:49:33.571 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:49:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:49:33.282+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:49:33.571 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:49:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:49:33.282+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:49:33.571 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:49:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:49:33.282+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:49:33.571 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:49:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:49:33.283+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:49:34.170 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:49:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:49:33.916+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:49:34.171 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:49:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:49:33.917+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:49:34.171 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:49:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:49:33.917+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:49:34.171 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:49:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:49:33.917+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:49:34.570 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:49:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:49:34.169+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:49:34.570 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:49:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:49:34.169+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:49:34.570 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:49:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:49:34.169+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:49:34.571 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:49:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:49:34.169+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:49:34.571 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:49:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:49:34.302+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:49:34.571 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:49:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:49:34.302+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:49:34.571 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:49:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:49:34.302+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:49:34.571 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:49:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:49:34.302+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:49:35.218 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:49:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:49:34.940+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:49:35.219 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:49:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:49:34.940+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:49:35.219 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:49:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:49:34.940+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:49:35.219 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:49:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:49:34.941+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:49:35.570 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:49:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:49:35.217+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:49:35.570 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:49:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:49:35.217+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:49:35.571 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:49:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:49:35.217+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:49:35.571 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:49:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:49:35.217+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:49:35.571 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:49:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:49:35.342+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:49:35.571 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:49:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:49:35.342+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:49:35.571 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:49:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:49:35.342+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:49:35.571 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:49:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:49:35.342+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:49:35.903 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:49:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:49:35.632+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 26739 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:49:36.206 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:49:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:49:35.901+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:49:36.206 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:49:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:49:35.901+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:49:36.206 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:49:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:49:35.901+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:49:36.207 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:49:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:49:35.901+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:49:36.570 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:49:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:49:36.205+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:49:36.570 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:49:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:49:36.205+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:49:36.570 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:49:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:49:36.205+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:49:36.570 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:49:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:49:36.206+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:49:36.571 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:49:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:49:36.393+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:49:36.571 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:49:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:49:36.393+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:49:36.571 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:49:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:49:36.393+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:49:36.571 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:49:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:49:36.394+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:49:37.214 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:49:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:49:36.876+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:49:37.215 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:49:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:49:36.876+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:49:37.215 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:49:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:49:36.876+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:49:37.215 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:49:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:49:36.876+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:49:37.570 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:49:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:49:37.213+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:49:37.570 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:49:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:49:37.213+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:49:37.570 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:49:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:49:37.213+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:49:37.570 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:49:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:49:37.214+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:49:37.571 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:49:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:49:37.377+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:49:37.571 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:49:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:49:37.377+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:49:37.571 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:49:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:49:37.377+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:49:37.571 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:49:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:49:37.378+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:49:38.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:49:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:49:37.885+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:49:38.197 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:49:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:49:37.885+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:49:38.197 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:49:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:49:37.885+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:49:38.197 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:49:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:49:37.886+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:49:38.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:49:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:49:38.378+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:49:38.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:49:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:49:38.378+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:49:38.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:49:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:49:38.378+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:49:38.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:49:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:49:38.379+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:49:38.571 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:49:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:49:38.195+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:49:38.571 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:49:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:49:38.195+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:49:38.571 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:49:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:49:38.195+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:49:38.571 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:49:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:49:38.195+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:49:39.154 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:49:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:49:38.897+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:49:39.154 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:49:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:49:38.897+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:49:39.154 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:49:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:49:38.897+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:49:39.154 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:49:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:49:38.897+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:49:39.550 ERROR:paramiko.transport:Socket exception: No route to host (113) 2024-03-18T01:49:39.555 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T01:49:39.555 INFO:tasks.workunit:Stopping ['suites/fsstress.sh'] on client.1... 2024-03-18T01:49:39.556 ERROR:teuthology.run_tasks:Manager failed: kclient Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_batrick_ceph_c382027d5c66ef4e0aa1131ea3e8080c658d3184/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_batrick_ceph_c382027d5c66ef4e0aa1131ea3e8080c658d3184/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_batrick_ceph_c382027d5c66ef4e0aa1131ea3e8080c658d3184/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host 2024-03-18T01:49:39.568 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-03-18T01:49:39.579 INFO:tasks.cephadm:Teardown begin 2024-03-18T01:49:39.580 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_batrick_ceph_c382027d5c66ef4e0aa1131ea3e8080c658d3184/qa/tasks/cephadm.py", line 1867, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_batrick_ceph_c382027d5c66ef4e0aa1131ea3e8080c658d3184/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_batrick_ceph_c382027d5c66ef4e0aa1131ea3e8080c658d3184/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_batrick_ceph_c382027d5c66ef4e0aa1131ea3e8080c658d3184/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host 2024-03-18T01:49:39.585 DEBUG:teuthology.orchestra.run.smithi029:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-18T01:49:39.588 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:49:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:49:39.338+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:49:39.588 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:49:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:49:39.338+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:49:39.588 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:49:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:49:39.338+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:49:39.588 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:49:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:49:39.338+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:49:39.589 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:49:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:49:39.152+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:49:39.589 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:49:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:49:39.152+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:49:39.589 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:49:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:49:39.152+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:49:39.589 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:49:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:49:39.153+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:49:39.620 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2024-03-18T01:49:39.621 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-18T01:49:40.177 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:49:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:49:39.894+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:49:40.178 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:49:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:49:39.894+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:49:40.178 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:49:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:49:39.894+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:49:40.178 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:49:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:49:39.894+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:49:40.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:49:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:49:40.344+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:49:40.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:49:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:49:40.344+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:49:40.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:49:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:49:40.344+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:49:40.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:49:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:49:40.345+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:49:40.571 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:49:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:49:40.176+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:49:40.571 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:49:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:49:40.176+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:49:40.571 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:49:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:49:40.176+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:49:40.571 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:49:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:49:40.177+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:49:41.070 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:49:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:49:40.636+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 26821 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:49:41.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:49:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:49:40.857+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:49:41.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:49:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:49:40.857+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:49:41.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:49:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:49:40.857+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:49:41.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:49:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:49:40.857+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:49:41.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:49:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:49:41.345+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:49:41.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:49:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:49:41.345+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:49:41.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:49:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:49:41.345+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:49:41.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:49:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:49:41.346+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:49:41.570 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:49:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:49:41.143+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:49:41.571 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:49:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:49:41.143+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:49:41.571 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:49:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:49:41.143+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:49:41.571 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:49:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:49:41.144+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:49:42.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:49:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:49:41.895+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:49:42.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:49:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:49:41.895+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:49:42.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:49:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:49:41.895+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:49:42.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:49:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:49:41.895+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:49:42.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:49:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:49:42.318+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:49:42.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:49:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:49:42.318+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:49:42.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:49:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:49:42.318+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:49:42.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:49:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:49:42.319+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:49:42.571 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:49:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:49:42.155+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:49:42.571 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:49:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:49:42.155+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:49:42.571 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:49:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:49:42.155+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:49:42.571 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:49:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:49:42.156+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:49:42.686 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.90 2024-03-18T01:49:43.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:49:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:49:43.144+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:49:43.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:49:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:49:43.144+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:49:43.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:49:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:49:43.144+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:49:43.321 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:49:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:49:43.145+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:49:43.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:49:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:49:42.922+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:49:43.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:49:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:49:42.922+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:49:43.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:49:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:49:42.922+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:49:43.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:49:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:49:42.922+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:49:43.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:49:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:49:43.328+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:49:43.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:49:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:49:43.328+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:49:43.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:49:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:49:43.328+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:49:43.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:49:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:49:43.328+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:49:44.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:49:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:49:44.098+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:49:44.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:49:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:49:44.098+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:49:44.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:49:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:49:44.098+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:49:44.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:49:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:49:44.098+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:49:44.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:49:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:49:43.955+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:49:44.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:49:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:49:43.955+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:49:44.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:49:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:49:43.955+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:49:44.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:49:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:49:43.956+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:49:44.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:49:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:49:44.372+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:49:44.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:49:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:49:44.372+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:49:44.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:49:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:49:44.372+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:49:44.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:49:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:49:44.372+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:49:45.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:49:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:49:45.109+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:49:45.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:49:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:49:45.109+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:49:45.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:49:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:49:45.109+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:49:45.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:49:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:49:45.109+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:49:45.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:49:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:49:44.934+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:49:45.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:49:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:49:44.934+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:49:45.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:49:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:49:44.934+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:49:45.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:49:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:49:44.935+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:49:45.762 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:49:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:49:45.639+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 26906 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:49:45.763 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:49:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:49:45.413+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:49:45.763 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:49:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:49:45.413+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:49:45.763 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:49:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:49:45.413+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:49:45.763 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:49:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:49:45.414+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:49:46.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:49:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:49:45.935+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:49:46.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:49:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:49:45.935+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:49:46.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:49:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:49:45.935+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:49:46.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:49:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:49:45.935+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:49:46.418 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:49:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:49:46.078+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:49:46.418 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:49:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:49:46.078+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:49:46.418 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:49:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:49:46.078+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:49:46.418 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:49:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:49:46.080+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:49:46.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:49:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:49:46.416+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:49:46.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:49:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:49:46.416+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:49:46.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:49:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:49:46.416+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:49:46.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:49:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:49:46.417+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:49:47.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:49:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:49:46.955+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:49:47.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:49:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:49:46.955+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:49:47.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:49:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:49:46.955+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:49:47.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:49:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:49:46.956+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:49:47.321 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:49:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:49:47.086+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:49:47.321 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:49:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:49:47.086+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:49:47.321 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:49:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:49:47.086+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:49:47.321 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:49:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:49:47.087+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:49:47.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:49:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:49:47.415+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:49:47.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:49:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:49:47.415+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:49:47.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:49:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:49:47.415+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:49:47.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:49:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:49:47.416+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:49:48.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:49:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:49:48.082+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:49:48.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:49:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:49:48.082+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:49:48.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:49:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:49:48.082+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:49:48.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:49:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:49:48.083+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:49:48.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:49:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:49:47.963+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:49:48.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:49:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:49:47.963+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:49:48.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:49:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:49:47.963+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:49:48.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:49:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:49:47.963+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:49:48.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:49:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:49:48.368+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:49:48.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:49:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:49:48.368+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:49:48.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:49:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:49:48.368+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:49:48.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:49:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:49:48.369+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:49:49.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:49:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:49:49.008+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:49:49.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:49:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:49:49.008+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:49:49.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:49:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:49:49.008+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:49:49.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:49:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:49:49.008+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:49:49.321 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:49:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:49:49.074+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:49:49.321 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:49:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:49:49.074+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:49:49.321 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:49:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:49:49.074+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:49:49.321 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:49:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:49:49.075+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:49:49.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:49:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:49:49.368+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:49:49.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:49:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:49:49.368+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:49:49.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:49:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:49:49.368+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:49:49.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:49:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:49:49.368+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:49:50.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:49:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:49:50.039+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:49:50.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:49:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:49:50.039+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:49:50.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:49:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:49:50.039+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:49:50.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:49:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:49:50.039+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:49:50.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:49:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:49:50.035+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:49:50.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:49:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:49:50.035+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:49:50.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:49:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:49:50.035+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:49:50.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:49:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:49:50.036+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:49:50.643 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:49:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:49:50.355+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:49:50.644 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:49:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:49:50.355+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:49:50.644 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:49:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:49:50.355+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:49:50.644 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:49:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:49:50.355+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:49:50.992 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:49:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:49:50.642+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 26988 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:49:51.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:49:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:49:50.991+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:49:51.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:49:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:49:50.991+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:49:51.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:49:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:49:50.991+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:49:51.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:49:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:49:50.991+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:49:51.321 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:49:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:49:51.029+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:49:51.321 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:49:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:49:51.029+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:49:51.321 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:49:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:49:51.029+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:49:51.321 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:49:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:49:51.030+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:49:51.687 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2024-03-18T01:49:51.688 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-18T01:49:51.737 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:49:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:49:51.385+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:49:51.738 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:49:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:49:51.385+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:49:51.738 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:49:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:49:51.385+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:49:51.738 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:49:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:49:51.386+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:49:52.027 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:49:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:49:51.979+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:49:52.027 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:49:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:49:51.979+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:49:52.027 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:49:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:49:51.979+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:49:52.027 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:49:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:49:51.979+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:49:52.283 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:49:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:49:52.025+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:49:52.284 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:49:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:49:52.025+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:49:52.284 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:49:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:49:52.025+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:49:52.284 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:49:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:49:52.026+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:49:52.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:49:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:49:52.352+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:49:52.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:49:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:49:52.352+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:49:52.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:49:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:49:52.352+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:49:52.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:49:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:49:52.353+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:49:53.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:49:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:49:53.026+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:49:53.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:49:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:49:53.026+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:49:53.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:49:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:49:53.026+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:49:53.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:49:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:49:53.027+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:49:53.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:49:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:49:53.018+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:49:53.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:49:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:49:53.018+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:49:53.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:49:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:49:53.018+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:49:53.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:49:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:49:53.019+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:49:53.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:49:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:49:53.397+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:49:53.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:49:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:49:53.397+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:49:53.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:49:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:49:53.397+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:49:53.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:49:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:49:53.397+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:49:54.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:49:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:49:54.006+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:49:54.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:49:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:49:54.006+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:49:54.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:49:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:49:54.006+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:49:54.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:49:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:49:54.007+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:49:54.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:49:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:49:53.981+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:49:54.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:49:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:49:53.981+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:49:54.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:49:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:49:53.981+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:49:54.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:49:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:49:53.981+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:49:54.754 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.90 2024-03-18T01:49:54.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:49:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:49:54.414+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:49:54.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:49:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:49:54.415+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:49:54.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:49:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:49:54.415+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:49:54.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:49:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:49:54.415+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:49:55.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:49:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:49:55.027+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:49:55.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:49:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:49:55.027+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:49:55.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:49:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:49:55.027+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:49:55.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:49:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:49:55.027+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:49:55.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:49:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:49:55.006+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:49:55.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:49:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:49:55.006+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:49:55.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:49:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:49:55.006+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:49:55.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:49:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:49:55.006+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:49:55.768 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:49:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:49:55.645+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 27073 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:49:55.768 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:49:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:49:55.427+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:49:55.768 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:49:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:49:55.427+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:49:55.768 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:49:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:49:55.427+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:49:55.768 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:49:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:49:55.428+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:49:56.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:49:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:49:56.060+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:49:56.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:49:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:49:56.060+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:49:56.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:49:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:49:56.060+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:49:56.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:49:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:49:56.060+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:49:56.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:49:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:49:56.049+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:49:56.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:49:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:49:56.049+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:49:56.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:49:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:49:56.049+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:49:56.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:49:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:49:56.050+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:49:56.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:49:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:49:56.402+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:49:56.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:49:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:49:56.402+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:49:56.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:49:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:49:56.402+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:49:56.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:49:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:49:56.403+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:49:57.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:49:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:49:57.069+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:49:57.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:49:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:49:57.069+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:49:57.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:49:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:49:57.069+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:49:57.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:49:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:49:57.070+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:49:57.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:49:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:49:57.021+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:49:57.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:49:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:49:57.021+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:49:57.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:49:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:49:57.021+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:49:57.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:49:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:49:57.022+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:49:57.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:49:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:49:57.379+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:49:57.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:49:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:49:57.379+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:49:57.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:49:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:49:57.379+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:49:57.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:49:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:49:57.380+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:49:58.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:49:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:49:57.986+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:49:58.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:49:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:49:57.986+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:49:58.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:49:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:49:57.986+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:49:58.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:49:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:49:57.986+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:49:58.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:49:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:49:58.082+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:49:58.321 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:49:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:49:58.082+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:49:58.321 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:49:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:49:58.082+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:49:58.321 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:49:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:49:58.083+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:49:58.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:49:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:49:58.338+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:49:58.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:49:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:49:58.338+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:49:58.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:49:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:49:58.338+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:49:58.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:49:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:49:58.338+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:49:59.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:49:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:49:59.048+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:49:59.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:49:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:49:59.048+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:49:59.321 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:49:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:49:59.048+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:49:59.321 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:49:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:49:59.049+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:49:59.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:49:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:49:59.003+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:49:59.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:49:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:49:59.003+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:49:59.322 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:49:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:49:59.003+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:49:59.322 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:49:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:49:59.003+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:49:59.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:49:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:49:59.344+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:49:59.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:49:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:49:59.344+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:49:59.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:49:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:49:59.344+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:49:59.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:49:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:49:59.344+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:50:00.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:50:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:50:00.015+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:50:00.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:50:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:50:00.015+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:50:00.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:50:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:50:00.015+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:50:00.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:50:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:50:00.015+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:50:00.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:50:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:50:00.037+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:50:00.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:50:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:50:00.037+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:50:00.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:50:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:50:00.037+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:50:00.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:50:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:50:00.037+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:50:00.650 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:50:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:50:00.333+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:50:00.650 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:50:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:50:00.333+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:50:00.650 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:50:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:50:00.333+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:50:00.650 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:50:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:50:00.334+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:50:00.988 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:50:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:50:00.648+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 27155 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:50:01.298 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:50:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:50:01.013+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:50:01.298 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:50:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:50:01.013+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:50:01.299 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:50:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:50:01.013+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:50:01.299 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:50:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:50:01.014+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:50:01.299 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:50:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:50:00.987+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:50:01.299 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:50:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:50:00.987+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:50:01.299 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:50:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:50:00.987+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:50:01.299 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:50:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:50:00.988+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:50:01.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:50:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:50:01.297+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:50:01.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:50:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:50:01.297+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:50:01.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:50:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:50:01.297+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:50:01.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:50:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:50:01.297+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:50:02.016 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:50:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:50:01.976+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:50:02.016 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:50:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:50:01.977+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:50:02.016 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:50:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:50:01.977+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:50:02.017 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:50:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:50:01.977+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:50:02.271 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:50:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:50:02.015+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:50:02.271 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:50:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:50:02.015+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:50:02.271 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:50:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:50:02.015+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:50:02.271 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:50:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:50:02.015+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:50:02.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:50:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:50:02.269+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:50:02.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:50:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:50:02.269+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:50:02.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:50:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:50:02.269+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:50:02.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:50:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:50:02.270+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:50:03.264 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:50:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:50:02.993+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:50:03.264 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:50:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:50:02.993+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:50:03.265 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:50:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:50:02.993+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:50:03.265 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:50:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:50:02.994+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:50:03.265 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:50:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:50:02.932+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:50:03.265 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:50:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:50:02.932+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:50:03.265 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:50:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:50:02.932+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:50:03.266 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:50:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:50:02.933+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:50:03.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:50:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:50:03.261+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:50:03.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:50:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:50:03.261+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:50:03.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:50:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:50:03.262+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:50:03.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:50:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:50:03.262+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:50:04.285 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:50:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:50:03.985+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:50:04.285 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:50:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:50:03.985+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:50:04.285 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:50:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:50:03.985+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:50:04.286 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:50:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:50:03.986+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:50:04.286 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:50:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:50:03.900+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:50:04.286 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:50:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:50:03.901+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:50:04.286 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:50:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:50:03.901+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:50:04.286 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:50:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:50:03.901+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:50:04.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:50:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:50:04.282+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:50:04.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:50:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:50:04.282+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:50:04.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:50:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:50:04.282+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:50:04.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:50:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:50:04.283+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:50:05.306 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:50:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:50:04.970+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:50:05.306 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:50:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:50:04.970+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:50:05.307 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:50:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:50:04.970+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:50:05.307 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:50:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:50:04.970+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:50:05.307 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:50:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:50:04.890+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:50:05.307 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:50:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:50:04.890+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:50:05.307 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:50:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:50:04.890+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:50:05.307 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:50:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:50:04.891+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:50:05.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:50:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:50:05.305+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:50:05.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:50:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:50:05.305+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:50:05.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:50:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:50:05.305+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:50:05.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:50:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:50:05.306+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:50:05.930 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:50:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:50:05.651+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 27240 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:50:06.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:50:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:50:05.929+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:50:06.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:50:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:50:05.929+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:50:06.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:50:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:50:05.929+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:50:06.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:50:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:50:05.930+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:50:06.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:50:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:50:06.016+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:50:06.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:50:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:50:06.016+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:50:06.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:50:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:50:06.016+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:50:06.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:50:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:50:06.017+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:50:06.756 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2024-03-18T01:50:06.756 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-18T01:50:06.822 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:50:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:50:06.331+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:50:06.822 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:50:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:50:06.331+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:50:06.822 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:50:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:50:06.331+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:50:06.823 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:50:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:50:06.332+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:50:07.296 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:50:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:50:06.930+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:50:07.297 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:50:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:50:06.930+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:50:07.297 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:50:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:50:06.930+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:50:07.297 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:50:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:50:06.930+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:50:07.297 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:50:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:50:07.049+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:50:07.297 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:50:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:50:07.049+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:50:07.298 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:50:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:50:07.049+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:50:07.298 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:50:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:50:07.050+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:50:07.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:50:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:50:07.295+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:50:07.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:50:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:50:07.295+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:50:07.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:50:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:50:07.295+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:50:07.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:50:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:50:07.295+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:50:08.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:50:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:50:07.934+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:50:08.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:50:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:50:07.934+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:50:08.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:50:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:50:07.934+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:50:08.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:50:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:50:07.934+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:50:08.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:50:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:50:08.016+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:50:08.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:50:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:50:08.016+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:50:08.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:50:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:50:08.016+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:50:08.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:50:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:50:08.016+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:50:08.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:50:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:50:08.337+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:50:08.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:50:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:50:08.337+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:50:08.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:50:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:50:08.337+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:50:08.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:50:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:50:08.338+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:50:09.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:50:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:50:08.955+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:50:09.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:50:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:50:08.955+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:50:09.321 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:50:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:50:08.955+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:50:09.321 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:50:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:50:08.956+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:50:09.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:50:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:50:08.991+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:50:09.322 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:50:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:50:08.991+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:50:09.322 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:50:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:50:08.991+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:50:09.323 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:50:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:50:08.992+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:50:09.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:50:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:50:09.381+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:50:09.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:50:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:50:09.381+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:50:09.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:50:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:50:09.381+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:50:09.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:50:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:50:09.382+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:50:09.826 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.90 2024-03-18T01:50:10.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:50:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:50:09.964+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:50:10.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:50:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:50:09.964+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:50:10.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:50:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:50:09.964+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:50:10.321 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:50:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:50:09.964+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:50:10.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:50:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:50:10.017+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:50:10.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:50:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:50:10.017+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:50:10.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:50:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:50:10.017+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:50:10.322 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:50:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:50:10.018+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:50:10.656 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:50:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:50:10.371+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:50:10.656 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:50:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:50:10.371+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:50:10.656 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:50:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:50:10.371+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:50:10.656 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:50:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:50:10.372+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:50:10.995 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:50:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:50:10.654+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 27322 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:50:11.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:50:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:50:11.034+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:50:11.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:50:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:50:11.034+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:50:11.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:50:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:50:11.034+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:50:11.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:50:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:50:11.035+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:50:11.321 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:50:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:50:10.994+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:50:11.321 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:50:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:50:10.994+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:50:11.321 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:50:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:50:10.994+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:50:11.321 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:50:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:50:10.994+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:50:11.675 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:50:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:50:11.370+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:50:11.675 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:50:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:50:11.370+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:50:11.676 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:50:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:50:11.370+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:50:11.676 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:50:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:50:11.371+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:50:12.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:50:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:50:12.008+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:50:12.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:50:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:50:12.008+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:50:12.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:50:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:50:12.008+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:50:12.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:50:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:50:12.009+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:50:12.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:50:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:50:12.021+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:50:12.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:50:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:50:12.021+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:50:12.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:50:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:50:12.021+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:50:12.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:50:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:50:12.021+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:50:12.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:50:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:50:12.353+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:50:12.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:50:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:50:12.353+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:50:12.821 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:50:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:50:12.353+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:50:12.821 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:50:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:50:12.354+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:50:12.829 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2024-03-18T01:50:12.831 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-18T01:50:12.894 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.90 2024-03-18T01:50:12.894 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-03-18T01:50:12.895 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-03-18T01:50:12.895 DEBUG:teuthology.orchestra.run.smithi029:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-18T01:50:12.911 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2024-03-18T01:50:12.911 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-18T01:50:13.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:50:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:50:12.988+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:50:13.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:50:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:50:12.988+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:50:13.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:50:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:50:12.988+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:50:13.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:50:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:50:12.988+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:50:13.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:50:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:50:13.046+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:50:13.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:50:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:50:13.046+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:50:13.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:50:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:50:13.046+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:50:13.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:50:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:50:13.047+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:50:13.789 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:50:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:50:13.390+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:50:13.789 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:50:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:50:13.390+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:50:13.789 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:50:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:50:13.390+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:50:13.789 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:50:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:50:13.391+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:50:14.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:50:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:50:14.005+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:50:14.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:50:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:50:14.005+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:50:14.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:50:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:50:14.005+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:50:14.071 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:50:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:50:14.006+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:50:14.397 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:50:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:50:14.088+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:50:14.397 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:50:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:50:14.088+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:50:14.397 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:50:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:50:14.088+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:50:14.397 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:50:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:50:14.089+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:50:14.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:50:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:50:14.395+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:50:14.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:50:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:50:14.395+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:50:14.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:50:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:50:14.395+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:50:14.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:50:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:50:14.396+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:50:15.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:50:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:50:14.958+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:50:15.321 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:50:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:50:14.958+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:50:15.321 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:50:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:50:14.958+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:50:15.321 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:50:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:50:14.958+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:50:15.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:50:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:50:15.040+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:50:15.322 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:50:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:50:15.040+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:50:15.322 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:50:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:50:15.040+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:50:15.322 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:50:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:50:15.041+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:50:15.661 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:50:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:50:15.428+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:50:15.661 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:50:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:50:15.428+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:50:15.661 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:50:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:50:15.428+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:50:15.661 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:50:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:50:15.429+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:50:15.966 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.90 2024-03-18T01:50:15.998 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:50:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:50:15.657+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 27407 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:50:16.322 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:50:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:50:16.017+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:50:16.323 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:50:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:50:16.017+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:50:16.324 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:50:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:50:16.017+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:50:16.324 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:50:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:50:16.018+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:50:16.324 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:50:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:50:15.997+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:50:16.324 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:50:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:50:15.997+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:50:16.324 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:50:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:50:15.997+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:50:16.324 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:50:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:50:15.998+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:50:16.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:50:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:50:16.392+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:50:16.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:50:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:50:16.392+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:50:16.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:50:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:50:16.392+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:50:16.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:50:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:50:16.393+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:50:17.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:50:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:50:16.968+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:50:17.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:50:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:50:16.968+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:50:17.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:50:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:50:16.968+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:50:17.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:50:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:50:16.969+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:50:17.321 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:50:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:50:17.039+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:50:17.321 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:50:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:50:17.039+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:50:17.321 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:50:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:50:17.039+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:50:17.321 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:50:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:50:17.040+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:50:17.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:50:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:50:17.416+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:50:17.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:50:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:50:17.416+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:50:17.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:50:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:50:17.416+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:50:17.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:50:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:50:17.416+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:50:18.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:50:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:50:17.960+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:50:18.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:50:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:50:17.960+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:50:18.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:50:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:50:17.960+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:50:18.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:50:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:50:17.960+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:50:18.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:50:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:50:18.080+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:50:18.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:50:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:50:18.080+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:50:18.321 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:50:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:50:18.080+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:50:18.321 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:50:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:50:18.081+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:50:18.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:50:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:50:18.394+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:50:18.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:50:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:50:18.394+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:50:18.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:50:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:50:18.394+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:50:18.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:50:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:50:18.395+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:50:19.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:50:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:50:18.938+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:50:19.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:50:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:50:18.938+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:50:19.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:50:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:50:18.938+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:50:19.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:50:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:50:18.938+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:50:19.321 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:50:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:50:19.040+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:50:19.321 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:50:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:50:19.040+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:50:19.321 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:50:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:50:19.040+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:50:19.321 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:50:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:50:19.041+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:50:19.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:50:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:50:19.430+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:50:19.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:50:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:50:19.430+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:50:19.821 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:50:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:50:19.430+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:50:19.821 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:50:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:50:19.432+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:50:20.308 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:50:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:50:19.975+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:50:20.309 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:50:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:50:19.975+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:50:20.309 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:50:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:50:19.975+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:50:20.309 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:50:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:50:19.976+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:50:20.309 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:50:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:50:20.035+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:50:20.309 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:50:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:50:20.035+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:50:20.310 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:50:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:50:20.035+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:50:20.310 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:50:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:50:20.036+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:50:20.559 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:50:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:50:20.455+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:50:20.559 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:50:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:50:20.455+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:50:20.560 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:50:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:50:20.455+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:50:20.560 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:50:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:50:20.456+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:50:20.820 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:50:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:50:20.660+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 27489 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:50:21.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:50:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:50:20.948+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:50:21.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:50:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:50:20.948+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:50:21.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:50:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:50:20.948+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:50:21.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:50:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:50:20.948+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:50:21.321 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:50:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:50:21.075+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:50:21.321 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:50:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:50:21.075+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:50:21.321 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:50:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:50:21.075+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:50:21.321 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:50:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:50:21.076+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:50:21.738 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:50:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:50:21.455+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:50:21.739 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:50:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:50:21.455+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:50:21.739 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:50:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:50:21.455+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:50:21.739 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:50:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:50:21.455+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:50:22.060 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:50:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:50:21.914+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:50:22.060 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:50:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:50:21.914+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:50:22.060 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:50:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:50:21.914+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:50:22.060 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:50:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:50:21.914+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:50:22.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:50:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:50:22.058+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:50:22.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:50:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:50:22.058+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:50:22.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:50:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:50:22.058+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:50:22.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:50:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:50:22.059+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:50:22.801 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:50:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:50:22.447+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:50:22.801 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:50:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:50:22.447+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:50:22.801 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:50:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:50:22.447+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:50:22.801 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:50:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:50:22.448+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:50:23.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:50:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:50:22.924+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:50:23.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:50:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:50:22.924+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:50:23.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:50:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:50:22.924+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:50:23.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:50:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:50:22.924+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:50:23.499 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:50:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:50:23.099+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:50:23.499 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:50:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:50:23.099+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:50:23.499 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:50:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:50:23.099+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:50:23.500 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:50:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:50:23.100+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:50:23.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:50:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:50:23.498+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:50:23.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:50:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:50:23.498+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:50:23.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:50:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:50:23.498+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:50:23.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:50:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:50:23.499+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:50:24.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:50:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:50:24.066+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:50:24.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:50:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:50:24.066+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:50:24.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:50:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:50:24.066+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:50:24.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:50:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:50:24.067+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:50:24.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:50:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:50:23.947+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:50:24.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:50:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:50:23.947+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:50:24.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:50:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:50:23.947+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:50:24.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:50:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:50:23.947+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:50:24.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:50:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:50:24.477+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:50:24.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:50:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:50:24.477+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:50:24.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:50:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:50:24.477+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:50:24.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:50:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:50:24.478+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:50:24.967 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2024-03-18T01:50:24.968 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-18T01:50:25.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:50:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:50:25.062+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:50:25.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:50:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:50:25.062+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:50:25.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:50:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:50:25.062+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:50:25.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:50:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:50:25.062+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:50:25.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:50:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:50:24.916+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:50:25.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:50:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:50:24.916+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:50:25.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:50:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:50:24.916+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:50:25.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:50:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:50:24.917+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:50:25.789 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:50:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:50:25.662+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 27574 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:50:25.789 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:50:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:50:25.478+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:50:25.789 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:50:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:50:25.478+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:50:25.789 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:50:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:50:25.478+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:50:25.789 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:50:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:50:25.479+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:50:26.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:50:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:50:25.928+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:50:26.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:50:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:50:25.928+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:50:26.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:50:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:50:25.928+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:50:26.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:50:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:50:25.928+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:50:26.380 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:50:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:50:26.092+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:50:26.380 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:50:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:50:26.092+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:50:26.380 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:50:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:50:26.092+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:50:26.380 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:50:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:50:26.092+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:50:26.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:50:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:50:26.509+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:50:26.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:50:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:50:26.509+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:50:26.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:50:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:50:26.509+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:50:26.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:50:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:50:26.510+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:50:27.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:50:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:50:27.051+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:50:27.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:50:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:50:27.051+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:50:27.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:50:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:50:27.051+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:50:27.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:50:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:50:27.052+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:50:27.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:50:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:50:26.965+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:50:27.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:50:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:50:26.965+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:50:27.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:50:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:50:26.965+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:50:27.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:50:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:50:26.965+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:50:27.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:50:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:50:27.492+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:50:27.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:50:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:50:27.492+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:50:27.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:50:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:50:27.492+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:50:27.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:50:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:50:27.492+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:50:28.030 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.90 2024-03-18T01:50:28.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:50:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:50:27.976+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:50:28.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:50:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:50:27.976+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:50:28.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:50:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:50:27.976+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:50:28.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:50:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:50:27.977+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:50:28.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:50:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:50:28.066+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:50:28.321 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:50:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:50:28.066+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:50:28.321 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:50:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:50:28.066+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:50:28.321 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:50:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:50:28.067+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:50:28.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:50:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:50:28.533+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:50:28.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:50:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:50:28.533+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:50:28.821 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:50:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:50:28.533+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:50:28.821 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:50:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:50:28.534+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:50:29.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:50:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:50:29.029+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:50:29.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:50:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:50:29.029+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:50:29.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:50:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:50:29.029+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:50:29.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:50:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:50:29.030+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:50:29.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:50:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:50:29.001+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:50:29.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:50:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:50:29.001+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:50:29.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:50:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:50:29.001+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:50:29.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:50:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:50:29.002+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:50:29.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:50:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:50:29.493+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:50:29.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:50:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:50:29.493+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:50:29.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:50:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:50:29.493+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:50:29.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:50:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:50:29.494+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:50:30.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:50:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:50:30.030+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:50:30.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:50:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:50:30.030+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:50:30.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:50:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:50:30.030+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:50:30.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:50:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:50:30.030+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:50:30.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:50:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:50:29.997+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:50:30.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:50:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:50:29.997+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:50:30.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:50:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:50:29.997+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:50:30.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:50:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:50:29.997+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:50:30.790 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:50:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:50:30.666+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 27656 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:50:30.790 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:50:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:50:30.487+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:50:30.790 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:50:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:50:30.487+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:50:30.791 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:50:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:50:30.487+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:50:30.791 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:50:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:50:30.487+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:50:31.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:50:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:50:31.009+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:50:31.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:50:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:50:31.009+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:50:31.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:50:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:50:31.009+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:50:31.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:50:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:50:31.010+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:50:31.521 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:50:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:50:31.079+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:50:31.521 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:50:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:50:31.079+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:50:31.521 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:50:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:50:31.079+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:50:31.521 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:50:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:50:31.080+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:50:31.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:50:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:50:31.518+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:50:31.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:50:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:50:31.518+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:50:31.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:50:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:50:31.518+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:50:31.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:50:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:50:31.519+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:50:32.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:50:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:50:31.991+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:50:32.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:50:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:50:31.991+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:50:32.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:50:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:50:31.991+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:50:32.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:50:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:50:31.991+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:50:32.321 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:50:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:50:32.088+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:50:32.321 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:50:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:50:32.088+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:50:32.321 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:50:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:50:32.088+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:50:32.321 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:50:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:50:32.088+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:50:32.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:50:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:50:32.544+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:50:32.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:50:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:50:32.544+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:50:32.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:50:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:50:32.544+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:50:32.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:50:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:50:32.545+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:50:33.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:50:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:50:33.064+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:50:33.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:50:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:50:33.064+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:50:33.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:50:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:50:33.064+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:50:33.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:50:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:50:33.065+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:50:33.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:50:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:50:32.987+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:50:33.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:50:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:50:32.987+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:50:33.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:50:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:50:32.987+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:50:33.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:50:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:50:32.988+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:50:33.978 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:50:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:50:33.591+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:50:33.979 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:50:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:50:33.591+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:50:33.979 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:50:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:50:33.591+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:50:33.979 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:50:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:50:33.591+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:50:34.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:50:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:50:34.060+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:50:34.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:50:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:50:34.060+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:50:34.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:50:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:50:34.060+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:50:34.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:50:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:50:34.060+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:50:34.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:50:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:50:33.976+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:50:34.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:50:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:50:33.976+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:50:34.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:50:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:50:33.976+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:50:34.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:50:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:50:33.976+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:50:34.935 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:50:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:50:34.636+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:50:34.935 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:50:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:50:34.636+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:50:34.935 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:50:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:50:34.636+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:50:34.935 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:50:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:50:34.637+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:50:35.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:50:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:50:35.030+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:50:35.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:50:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:50:35.030+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:50:35.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:50:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:50:35.030+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:50:35.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:50:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:50:35.030+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:50:35.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:50:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:50:34.933+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:50:35.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:50:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:50:34.933+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:50:35.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:50:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:50:34.933+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:50:35.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:50:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:50:34.934+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:50:36.053 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:50:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:50:35.670+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 27741 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:50:36.053 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:50:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:50:35.651+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:50:36.054 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:50:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:50:35.651+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:50:36.054 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:50:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:50:35.651+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:50:36.054 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:50:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:50:35.652+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:50:36.054 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:50:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:50:35.884+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:50:36.054 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:50:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:50:35.884+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:50:36.054 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:50:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:50:35.884+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:50:36.055 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:50:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:50:35.885+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:50:36.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:50:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:50:36.051+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:50:36.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:50:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:50:36.051+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:50:36.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:50:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:50:36.051+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:50:36.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:50:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:50:36.052+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:50:36.866 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:50:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:50:36.603+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:50:36.867 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:50:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:50:36.603+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:50:36.867 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:50:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:50:36.603+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:50:36.867 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:50:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:50:36.604+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:50:37.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:50:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:50:37.077+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:50:37.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:50:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:50:37.077+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:50:37.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:50:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:50:37.077+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:50:37.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:50:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:50:37.078+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:50:37.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:50:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:50:36.865+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:50:37.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:50:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:50:36.865+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:50:37.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:50:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:50:36.865+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:50:37.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:50:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:50:36.865+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:50:38.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:50:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:50:37.647+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:50:38.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:50:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:50:37.647+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:50:38.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:50:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:50:37.647+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:50:38.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:50:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:50:37.648+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:50:38.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:50:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:50:37.876+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:50:38.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:50:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:50:37.876+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:50:38.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:50:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:50:37.876+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:50:38.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:50:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:50:37.876+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:50:38.570 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:50:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:50:38.098+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:50:38.570 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:50:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:50:38.098+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:50:38.570 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:50:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:50:38.098+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:50:38.570 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:50:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:50:38.099+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:50:39.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:50:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:50:38.661+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:50:39.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:50:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:50:38.661+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:50:39.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:50:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:50:38.661+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:50:39.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:50:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:50:38.661+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:50:39.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:50:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:50:38.857+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:50:39.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:50:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:50:38.857+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:50:39.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:50:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:50:38.857+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:50:39.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:50:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:50:38.857+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:50:39.570 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:50:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:50:39.119+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:50:39.570 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:50:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:50:39.119+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:50:39.570 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:50:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:50:39.119+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:50:39.570 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:50:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:50:39.120+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:50:40.031 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2024-03-18T01:50:40.032 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-18T01:50:40.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:50:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:50:39.670+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:50:40.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:50:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:50:39.670+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:50:40.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:50:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:50:39.670+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:50:40.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:50:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:50:39.671+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:50:40.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:50:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:50:39.898+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:50:40.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:50:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:50:39.898+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:50:40.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:50:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:50:39.898+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:50:40.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:50:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:50:39.899+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:50:40.570 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:50:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:50:40.096+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:50:40.570 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:50:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:50:40.096+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:50:40.570 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:50:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:50:40.096+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:50:40.571 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:50:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:50:40.096+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:50:41.070 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:50:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:50:40.673+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 27823 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:50:41.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:50:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:50:40.679+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:50:41.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:50:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:50:40.679+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:50:41.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:50:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:50:40.679+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:50:41.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:50:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:50:40.680+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:50:41.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:50:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:50:40.853+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:50:41.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:50:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:50:40.853+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:50:41.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:50:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:50:40.853+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:50:41.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:50:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:50:40.853+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:50:41.435 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:50:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:50:41.093+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:50:41.435 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:50:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:50:41.093+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:50:41.436 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:50:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:50:41.093+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:50:41.436 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:50:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:50:41.094+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:50:41.740 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:50:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:50:41.633+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:50:41.740 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:50:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:50:41.633+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:50:41.740 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:50:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:50:41.633+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:50:41.740 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:50:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:50:41.634+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:50:42.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:50:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:50:41.806+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:50:42.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:50:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:50:41.806+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:50:42.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:50:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:50:41.806+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:50:42.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:50:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:50:41.806+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:50:42.570 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:50:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:50:42.139+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:50:42.570 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:50:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:50:42.139+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:50:42.570 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:50:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:50:42.139+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:50:42.570 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:50:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:50:42.140+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:50:43.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:50:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:50:42.790+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:50:43.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:50:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:50:42.790+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:50:43.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:50:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:50:42.790+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:50:43.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:50:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:50:42.790+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:50:43.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:50:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:50:42.636+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:50:43.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:50:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:50:42.636+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:50:43.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:50:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:50:42.636+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:50:43.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:50:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:50:42.637+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:50:43.102 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.90 2024-03-18T01:50:43.570 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:50:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:50:43.188+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:50:43.570 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:50:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:50:43.188+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:50:43.570 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:50:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:50:43.188+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:50:43.570 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:50:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:50:43.189+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:50:44.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:50:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:50:43.599+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:50:44.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:50:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:50:43.599+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:50:44.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:50:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:50:43.599+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:50:44.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:50:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:50:43.600+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:50:44.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:50:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:50:43.779+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:50:44.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:50:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:50:43.779+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:50:44.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:50:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:50:43.779+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:50:44.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:50:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:50:43.779+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:50:44.507 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:50:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:50:44.149+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:50:44.508 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:50:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:50:44.149+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:50:44.508 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:50:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:50:44.149+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:50:44.508 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:50:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:50:44.150+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:50:44.816 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:50:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:50:44.578+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:50:44.817 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:50:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:50:44.578+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:50:44.817 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:50:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:50:44.578+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:50:44.817 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:50:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:50:44.578+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:50:45.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:50:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:50:44.815+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:50:45.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:50:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:50:44.815+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:50:45.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:50:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:50:44.815+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:50:45.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:50:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:50:44.816+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:50:45.570 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:50:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:50:45.114+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:50:45.570 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:50:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:50:45.114+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:50:45.570 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:50:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:50:45.114+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:50:45.570 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:50:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:50:45.116+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:50:46.070 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:50:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:50:45.676+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 27907 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:50:46.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:50:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:50:45.602+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:50:46.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:50:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:50:45.602+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:50:46.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:50:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:50:45.602+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:50:46.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:50:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:50:45.602+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:50:46.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:50:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:50:45.843+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:50:46.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:50:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:50:45.843+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:50:46.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:50:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:50:45.843+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:50:46.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:50:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:50:45.844+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:50:46.103 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2024-03-18T01:50:46.104 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-18T01:50:46.174 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.90 2024-03-18T01:50:46.174 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-03-18T01:50:46.175 INFO:tasks.cephadm:Archiving crash dumps... 2024-03-18T01:50:46.177 DEBUG:teuthology.misc:Transferring archived files from smithi029:/var/lib/ceph/37f51182-e4c4-11ee-95c9-87774f69a715/crash to /home/teuthworker/archive/pdonnell-2024-03-17_23:17:27-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7608561/remote/smithi029/crash 2024-03-18T01:50:46.178 DEBUG:teuthology.orchestra.run.smithi029:> sudo tar c -f - -C /var/lib/ceph/37f51182-e4c4-11ee-95c9-87774f69a715/crash -- . 2024-03-18T01:50:46.248 DEBUG:teuthology.misc:Transferring archived files from smithi090:/var/lib/ceph/37f51182-e4c4-11ee-95c9-87774f69a715/crash to /home/teuthworker/archive/pdonnell-2024-03-17_23:17:27-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7608561/remote/smithi090/crash 2024-03-18T01:50:46.249 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2024-03-18T01:50:46.250 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-18T01:50:46.561 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:50:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:50:46.079+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:50:46.561 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:50:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:50:46.079+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:50:46.561 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:50:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:50:46.079+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:50:46.562 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:50:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:50:46.079+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:50:46.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:50:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:50:46.559+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:50:46.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:50:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:50:46.559+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:50:46.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:50:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:50:46.559+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:50:46.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:50:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:50:46.560+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:50:47.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:50:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:50:47.090+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:50:47.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:50:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:50:47.090+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:50:47.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:50:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:50:47.090+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:50:47.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:50:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:50:47.090+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:50:47.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:50:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:50:46.888+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:50:47.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:50:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:50:46.888+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:50:47.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:50:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:50:46.888+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:50:47.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:50:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:50:46.889+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:50:47.909 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:50:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:50:47.589+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:50:47.910 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:50:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:50:47.589+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:50:47.910 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:50:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:50:47.589+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:50:47.910 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:50:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:50:47.590+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:50:48.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:50:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:50:48.059+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:50:48.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:50:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:50:48.059+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:50:48.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:50:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:50:48.059+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:50:48.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:50:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:50:48.059+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:50:48.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:50:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:50:47.908+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:50:48.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:50:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:50:47.908+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:50:48.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:50:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:50:47.908+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:50:48.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:50:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:50:47.908+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:50:48.936 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:50:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:50:48.590+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:50:48.936 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:50:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:50:48.590+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:50:48.936 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:50:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:50:48.590+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:50:48.936 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:50:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:50:48.591+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:50:49.314 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.90 2024-03-18T01:50:49.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:50:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:50:49.042+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:50:49.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:50:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:50:49.042+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:50:49.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:50:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:50:49.042+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:50:49.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:50:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:50:49.043+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:50:49.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:50:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:50:48.934+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:50:49.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:50:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:50:48.934+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:50:49.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:50:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:50:48.934+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:50:49.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:50:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:50:48.935+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:50:49.967 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:50:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:50:49.616+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:50:49.968 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:50:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:50:49.616+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:50:49.968 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:50:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:50:49.616+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:50:49.968 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:50:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:50:49.617+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:50:50.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:50:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:50:50.066+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:50:50.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:50:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:50:50.066+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:50:50.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:50:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:50:50.066+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:50:50.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:50:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:50:50.066+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:50:50.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:50:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:50:49.966+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:50:50.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:50:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:50:49.966+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:50:50.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:50:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:50:49.966+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:50:50.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:50:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:50:49.966+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:50:50.971 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:50:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:50:50.678+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 27990 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:50:50.972 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:50:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:50:50.619+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:50:50.972 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:50:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:50:50.619+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:50:50.972 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:50:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:50:50.619+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:50:50.972 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:50:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:50:50.620+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:50:51.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:50:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:50:50.970+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:50:51.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:50:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:50:50.970+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:50:51.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:50:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:50:50.970+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:50:51.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:50:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:50:50.970+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:50:51.321 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:50:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:50:51.062+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:50:51.321 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:50:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:50:51.062+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:50:51.321 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:50:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:50:51.062+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:50:51.321 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:50:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:50:51.063+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:50:52.021 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:50:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:50:51.628+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:50:52.021 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:50:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:50:51.628+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:50:52.021 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:50:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:50:51.628+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:50:52.021 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:50:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:50:51.629+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:50:52.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:50:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:50:52.030+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:50:52.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:50:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:50:52.030+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:50:52.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:50:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:50:52.030+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:50:52.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:50:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:50:52.031+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:50:52.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:50:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:50:52.019+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:50:52.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:50:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:50:52.019+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:50:52.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:50:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:50:52.019+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:50:52.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:50:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:50:52.020+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:50:53.003 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:50:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:50:52.629+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:50:53.003 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:50:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:50:52.629+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:50:53.003 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:50:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:50:52.629+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:50:53.003 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:50:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:50:52.630+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:50:53.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:50:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:50:53.019+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:50:53.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:50:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:50:53.019+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:50:53.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:50:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:50:53.019+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:50:53.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:50:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:50:53.019+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:50:53.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:50:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:50:53.001+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:50:53.321 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:50:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:50:53.001+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:50:53.321 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:50:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:50:53.001+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:50:53.321 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:50:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:50:53.002+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:50:53.984 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:50:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:50:53.678+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:50:53.984 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:50:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:50:53.678+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:50:53.984 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:50:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:50:53.678+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:50:53.984 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:50:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:50:53.679+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:50:54.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:50:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:50:54.014+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:50:54.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:50:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:50:54.014+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:50:54.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:50:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:50:54.014+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:50:54.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:50:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:50:54.015+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:50:54.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:50:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:50:53.982+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:50:54.321 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:50:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:50:53.982+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:50:54.321 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:50:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:50:53.982+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:50:54.321 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:50:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:50:53.984+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:50:54.981 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:50:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:50:54.665+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:50:54.981 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:50:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:50:54.665+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:50:54.981 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:50:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:50:54.665+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:50:54.981 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:50:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:50:54.666+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:50:55.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:50:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:50:54.980+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:50:55.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:50:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:50:54.980+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:50:55.321 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:50:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:50:54.980+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:50:55.321 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:50:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:50:54.981+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:50:55.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:50:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:50:54.979+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:50:55.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:50:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:50:54.979+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:50:55.322 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:50:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:50:54.979+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:50:55.322 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:50:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:50:54.980+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:50:55.970 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:50:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:50:55.681+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 28074 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:50:55.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:50:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:50:55.664+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:50:55.971 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:50:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:50:55.664+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:50:55.971 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:50:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:50:55.664+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:50:55.971 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:50:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:50:55.665+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:50:56.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:50:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:50:55.991+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:50:56.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:50:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:50:55.991+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:50:56.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:50:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:50:55.991+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:50:56.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:50:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:50:55.992+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:50:56.321 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:50:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:50:55.969+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:50:56.322 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:50:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:50:55.969+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:50:56.322 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:50:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:50:55.969+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:50:56.322 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:50:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:50:55.969+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:50:56.964 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:50:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:50:56.699+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:50:56.965 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:50:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:50:56.699+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:50:56.965 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:50:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:50:56.699+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:50:56.965 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:50:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:50:56.700+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:50:57.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:50:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:50:57.021+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:50:57.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:50:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:50:57.021+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:50:57.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:50:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:50:57.021+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:50:57.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:50:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:50:57.022+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:50:57.321 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:50:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:50:56.963+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:50:57.321 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:50:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:50:56.963+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:50:57.321 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:50:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:50:56.963+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:50:57.321 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:50:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:50:56.964+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:50:57.990 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:50:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:50:57.705+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:50:57.990 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:50:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:50:57.705+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:50:57.990 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:50:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:50:57.705+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:50:57.991 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:50:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:50:57.706+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:50:58.315 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2024-03-18T01:50:58.316 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-18T01:50:58.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:50:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:50:58.001+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:50:58.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:50:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:50:58.001+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:50:58.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:50:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:50:58.001+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:50:58.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:50:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:50:58.001+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:50:58.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:50:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:50:57.988+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:50:58.321 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:50:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:50:57.988+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:50:58.321 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:50:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:50:57.988+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:50:58.321 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:50:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:50:57.989+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:50:58.994 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:50:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:50:58.724+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:50:58.994 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:50:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:50:58.724+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:50:58.995 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:50:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:50:58.724+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:50:58.995 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:50:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:50:58.724+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:50:59.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:50:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:50:58.993+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:50:59.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:50:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:50:58.993+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:50:59.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:50:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:50:58.993+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:50:59.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:50:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:50:58.994+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:50:59.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:50:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:50:59.047+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:50:59.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:50:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:50:59.047+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:50:59.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:50:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:50:59.047+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:50:59.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:50:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:50:59.047+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:51:00.037 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:50:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:50:59.770+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:51:00.037 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:50:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:50:59.770+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:51:00.038 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:50:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:50:59.770+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:51:00.038 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:50:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:50:59.771+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:51:00.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:51:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:51:00.065+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:51:00.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:51:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:51:00.065+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:51:00.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:51:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:51:00.065+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:51:00.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:51:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:51:00.065+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:51:00.321 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:51:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:51:00.035+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:51:00.321 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:51:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:51:00.035+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:51:00.321 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:51:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:51:00.035+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:51:00.321 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:51:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:51:00.036+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:51:00.996 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:51:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:51:00.685+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 28157 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:51:00.997 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:51:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:51:00.789+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:51:00.997 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:51:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:51:00.789+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:51:00.997 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:51:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:51:00.789+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:51:00.997 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:51:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:51:00.789+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:51:01.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:51:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:51:01.060+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:51:01.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:51:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:51:01.060+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:51:01.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:51:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:51:01.060+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:51:01.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:51:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:51:01.061+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:51:01.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:51:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:51:00.995+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:51:01.321 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:51:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:51:00.995+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:51:01.321 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:51:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:51:00.995+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:51:01.321 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:51:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:51:00.996+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:51:01.374 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.90 2024-03-18T01:51:02.042 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:51:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:51:01.769+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:51:02.042 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:51:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:51:01.769+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:51:02.042 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:51:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:51:01.769+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:51:02.043 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:51:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:51:01.769+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:51:02.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:51:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:51:02.041+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:51:02.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:51:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:51:02.041+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:51:02.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:51:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:51:02.041+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:51:02.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:51:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:51:02.042+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:51:02.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:51:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:51:02.041+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:51:02.321 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:51:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:51:02.041+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:51:02.321 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:51:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:51:02.041+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:51:02.321 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:51:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:51:02.042+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:51:03.064 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:51:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:51:03.007+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:51:03.064 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:51:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:51:03.007+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:51:03.064 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:51:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:51:03.007+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:51:03.064 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:51:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:51:03.007+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:51:03.065 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:51:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:51:02.776+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:51:03.065 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:51:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:51:02.776+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:51:03.065 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:51:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:51:02.776+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:51:03.065 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:51:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:51:02.777+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:51:03.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:51:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:51:03.062+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:51:03.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:51:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:51:03.062+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:51:03.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:51:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:51:03.062+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:51:03.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:51:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:51:03.063+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:51:03.996 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:51:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:51:03.727+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:51:03.996 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:51:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:51:03.727+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:51:03.996 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:51:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:51:03.727+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:51:03.996 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:51:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:51:03.728+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:51:04.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:51:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:51:04.096+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:51:04.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:51:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:51:04.096+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:51:04.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:51:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:51:04.096+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:51:04.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:51:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:51:04.096+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:51:04.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:51:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:51:03.994+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:51:04.321 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:51:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:51:03.994+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:51:04.321 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:51:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:51:03.994+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:51:04.321 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:51:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:51:03.995+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:51:04.989 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:51:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:51:04.721+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:51:04.989 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:51:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:51:04.721+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:51:04.989 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:51:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:51:04.721+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:51:04.989 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:51:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:51:04.721+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:51:05.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:51:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:51:05.101+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:51:05.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:51:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:51:05.101+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:51:05.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:51:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:51:05.101+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:51:05.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:51:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:51:05.102+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:51:05.321 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:51:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:51:04.987+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:51:05.321 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:51:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:51:04.987+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:51:05.321 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:51:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:51:04.987+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:51:05.322 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:51:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:51:04.988+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:51:05.962 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:51:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:51:05.688+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 28240 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:51:05.962 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:51:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:51:05.740+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:51:05.962 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:51:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:51:05.740+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:51:05.962 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:51:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:51:05.740+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:51:05.962 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:51:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:51:05.740+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:51:06.322 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:51:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:51:05.960+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:51:06.322 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:51:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:51:05.960+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:51:06.322 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:51:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:51:05.960+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:51:06.323 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:51:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:51:05.961+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:51:06.323 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:51:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:51:06.086+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:51:06.323 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:51:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:51:06.086+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:51:06.323 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:51:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:51:06.086+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:51:06.323 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:51:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:51:06.087+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:51:07.009 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:51:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:51:06.705+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:51:07.009 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:51:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:51:06.705+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:51:07.010 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:51:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:51:06.705+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:51:07.010 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:51:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:51:06.706+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:51:07.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:51:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:51:07.008+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:51:07.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:51:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:51:07.008+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:51:07.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:51:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:51:07.008+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:51:07.321 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:51:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:51:07.009+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:51:07.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:51:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:51:07.104+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:51:07.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:51:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:51:07.104+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:51:07.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:51:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:51:07.104+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:51:07.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:51:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:51:07.105+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:51:08.021 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:51:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:51:07.709+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:51:08.022 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:51:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:51:07.709+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:51:08.022 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:51:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:51:07.709+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:51:08.022 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:51:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:51:07.710+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:51:08.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:51:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:51:08.138+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:51:08.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:51:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:51:08.138+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:51:08.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:51:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:51:08.138+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:51:08.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:51:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:51:08.138+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:51:08.321 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:51:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:51:08.020+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:51:08.321 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:51:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:51:08.020+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:51:08.321 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:51:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:51:08.020+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:51:08.321 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:51:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:51:08.021+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:51:09.029 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:51:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:51:08.736+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:51:09.029 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:51:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:51:08.736+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:51:09.029 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:51:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:51:08.736+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:51:09.029 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:51:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:51:08.737+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:51:09.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:51:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:51:09.027+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:51:09.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:51:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:51:09.027+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:51:09.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:51:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:51:09.027+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:51:09.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:51:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:51:09.028+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:51:09.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:51:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:51:09.119+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:51:09.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:51:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:51:09.119+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:51:09.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:51:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:51:09.119+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:51:09.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:51:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:51:09.120+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:51:09.995 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:51:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:51:09.712+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:51:09.995 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:51:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:51:09.712+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:51:09.996 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:51:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:51:09.712+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:51:09.996 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:51:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:51:09.712+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:51:10.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:51:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:51:10.157+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:51:10.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:51:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:51:10.157+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:51:10.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:51:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:51:10.157+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:51:10.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:51:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:51:10.157+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:51:10.321 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:51:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:51:09.994+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:51:10.321 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:51:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:51:09.994+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:51:10.321 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:51:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:51:09.994+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:51:10.321 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:51:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:51:09.994+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:51:11.040 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:51:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:51:10.691+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 28323 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:51:11.040 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:51:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:51:10.707+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:51:11.040 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:51:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:51:10.707+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:51:11.041 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:51:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:51:10.707+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:51:11.041 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:51:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:51:10.708+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:51:11.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:51:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:51:11.038+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:51:11.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:51:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:51:11.038+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:51:11.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:51:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:51:11.038+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:51:11.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:51:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:51:11.039+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:51:11.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:51:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:51:11.174+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:51:11.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:51:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:51:11.174+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:51:11.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:51:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:51:11.174+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:51:11.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:51:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:51:11.175+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:51:12.060 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:51:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:51:11.752+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:51:12.060 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:51:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:51:11.752+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:51:12.060 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:51:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:51:11.752+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:51:12.060 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:51:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:51:11.753+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:51:12.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:51:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:51:12.058+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:51:12.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:51:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:51:12.058+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:51:12.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:51:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:51:12.058+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:51:12.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:51:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:51:12.059+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:51:12.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:51:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:51:12.156+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:51:12.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:51:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:51:12.156+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:51:12.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:51:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:51:12.156+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:51:12.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:51:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:51:12.156+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:51:13.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:51:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:51:12.741+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:51:13.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:51:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:51:12.741+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:51:13.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:51:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:51:12.741+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:51:13.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:51:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:51:12.742+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:51:13.375 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2024-03-18T01:51:13.376 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-18T01:51:13.570 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:51:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:51:13.095+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:51:13.570 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:51:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:51:13.095+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:51:13.570 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:51:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:51:13.095+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:51:13.570 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:51:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:51:13.096+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:51:13.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:51:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:51:13.164+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:51:13.571 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:51:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:51:13.164+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:51:13.571 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:51:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:51:13.164+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:51:13.571 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:51:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:51:13.164+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:51:14.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:51:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:51:13.756+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:51:14.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:51:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:51:13.756+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:51:14.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:51:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:51:13.756+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:51:14.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:51:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:51:13.756+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:51:14.385 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:51:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:51:14.107+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:51:14.385 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:51:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:51:14.107+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:51:14.385 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:51:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:51:14.107+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:51:14.385 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:51:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:51:14.107+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:51:14.386 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:51:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:51:14.166+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:51:14.386 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:51:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:51:14.167+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:51:14.386 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:51:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:51:14.167+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:51:14.386 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:51:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:51:14.167+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:51:15.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:51:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:51:14.802+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:51:15.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:51:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:51:14.802+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:51:15.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:51:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:51:14.802+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:39.847904+0000 (oldest deadline 2024-03-18T01:40:59.847904+0000) 2024-03-18T01:51:15.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:51:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:51:14.802+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:51:15.362 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:51:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:51:15.183+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:51:15.363 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:51:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:51:15.183+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:51:15.363 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:51:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:51:15.183+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:51:15.363 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:51:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:51:15.183+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:51:15.363 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:51:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:51:15.107+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:51:15.363 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:51:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:51:15.107+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:51:15.364 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:51:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:51:15.107+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:51:15.364 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:51:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:51:15.108+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:51:16.070 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:51:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:51:15.694+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 28406 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:51:16.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:51:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:51:15.821+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:51:16.446 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.90 2024-03-18T01:51:16.570 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:51:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:51:16.110+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:51:16.570 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:51:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:51:16.110+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:51:16.570 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:51:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:51:16.110+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:51:16.571 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:51:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:51:16.110+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:51:16.571 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:51:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:51:16.205+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:51:16.571 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:51:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:51:16.205+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:51:16.571 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:51:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:51:16.205+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:51:16.571 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:51:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:51:16.205+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:51:17.120 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:51:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:51:16.856+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:51:17.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:51:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:51:17.183+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:51:17.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:51:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:51:17.183+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:51:17.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:51:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:51:17.183+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:51:17.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:51:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:51:17.183+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:51:17.570 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:51:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:51:17.119+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:51:17.571 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:51:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:51:17.119+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:51:17.571 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:51:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:51:17.119+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:51:17.571 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:51:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:51:17.120+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:51:18.211 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:51:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:51:17.896+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:51:18.211 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:51:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:51:18.134+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:51:18.211 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:51:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:51:18.134+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:51:18.211 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:51:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:51:18.134+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:51:18.211 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:51:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:51:18.135+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:51:18.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:51:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:51:18.209+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:51:18.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:51:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:51:18.209+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:51:18.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:51:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:51:18.209+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:51:18.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:51:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:51:18.210+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:51:19.214 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:51:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:51:18.885+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:51:19.215 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:51:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:51:19.121+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:51:19.215 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:51:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:51:19.121+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:51:19.215 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:51:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:51:19.121+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:51:19.215 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:51:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:51:19.122+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:51:19.447 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2024-03-18T01:51:19.448 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-18T01:51:19.518 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.90 2024-03-18T01:51:19.518 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-03-18T01:51:19.519 INFO:tasks.cephadm:Checking cluster log for badness... 2024-03-18T01:51:19.519 DEBUG:teuthology.orchestra.run.smithi029:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/37f51182-e4c4-11ee-95c9-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v 'mons down, quorum' | egrep -v OSD_DOWN | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v MON_DOWN | head -n 1 2024-03-18T01:51:19.549 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:51:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:51:19.213+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:51:19.549 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:51:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:51:19.213+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:51:19.549 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:51:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:51:19.213+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:51:19.550 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:51:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:51:19.213+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:51:19.555 INFO:tasks.cephadm:Compressing logs... 2024-03-18T01:51:19.556 DEBUG:teuthology.orchestra.run.smithi029:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-18T01:51:19.598 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2024-03-18T01:51:19.599 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-18T01:51:19.622 INFO:teuthology.orchestra.run.smithi029.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-18T01:51:19.622 INFO:teuthology.orchestra.run.smithi029.stderr:: No such file or directory 2024-03-18T01:51:19.622 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/37f51182-e4c4-11ee-95c9-87774f69a715/ceph-mds.cephfs.smithi029.brxqeh.log 2024-03-18T01:51:19.623 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/37f51182-e4c4-11ee-95c9-87774f69a715/ceph-mon.smithi029.log 2024-03-18T01:51:19.623 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/37f51182-e4c4-11ee-95c9-87774f69a715/ceph.audit.log 2024-03-18T01:51:19.623 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/37f51182-e4c4-11ee-95c9-87774f69a715/ceph-mds.cephfs.smithi029.brxqeh.log: /var/log/ceph/37f51182-e4c4-11ee-95c9-87774f69a715/ceph-mon.smithi029.log: gzip -5 --verbose -- /var/log/ceph/37f51182-e4c4-11ee-95c9-87774f69a715/ceph-osd.1.log 2024-03-18T01:51:19.624 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/37f51182-e4c4-11ee-95c9-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/37f51182-e4c4-11ee-95c9-87774f69a715/ceph-mds.cephfs.smithi029.xuxvlr.log 2024-03-18T01:51:19.625 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/37f51182-e4c4-11ee-95c9-87774f69a715/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/37f51182-e4c4-11ee-95c9-87774f69a715/ceph-mgr.smithi029.qviwhl.log 2024-03-18T01:51:19.626 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/37f51182-e4c4-11ee-95c9-87774f69a715/ceph-mds.cephfs.smithi029.xuxvlr.log: /var/log/ceph/37f51182-e4c4-11ee-95c9-87774f69a715/ceph-mgr.smithi029.qviwhl.log: gzip -5 --verbose -- /var/log/ceph/37f51182-e4c4-11ee-95c9-87774f69a715/ceph-osd.0.log 2024-03-18T01:51:19.626 INFO:teuthology.orchestra.run.smithi029.stderr: 89.9% -- replaced with /var/log/ceph/37f51182-e4c4-11ee-95c9-87774f69a715/ceph.audit.log.gz 2024-03-18T01:51:19.626 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/37f51182-e4c4-11ee-95c9-87774f69a715/ceph.cephadm.log 2024-03-18T01:51:19.626 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/37f51182-e4c4-11ee-95c9-87774f69a715/ceph.log 2024-03-18T01:51:19.627 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/37f51182-e4c4-11ee-95c9-87774f69a715/ceph-osd.0.log: /var/log/ceph/37f51182-e4c4-11ee-95c9-87774f69a715/ceph.cephadm.log: 81.5% -- replaced with /var/log/ceph/37f51182-e4c4-11ee-95c9-87774f69a715/ceph.cephadm.log.gz 2024-03-18T01:51:19.627 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/37f51182-e4c4-11ee-95c9-87774f69a715/ceph-osd.2.log 2024-03-18T01:51:19.628 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/37f51182-e4c4-11ee-95c9-87774f69a715/ceph.log: 85.8% -- replaced with /var/log/ceph/37f51182-e4c4-11ee-95c9-87774f69a715/ceph.log.gz 2024-03-18T01:51:19.628 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/37f51182-e4c4-11ee-95c9-87774f69a715/ceph-volume.log 2024-03-18T01:51:19.630 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/37f51182-e4c4-11ee-95c9-87774f69a715/ceph-osd.2.log: 93.2% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-18T01:51:19.642 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/37f51182-e4c4-11ee-95c9-87774f69a715/ceph-volume.log: 90.9% -- replaced with /var/log/ceph/37f51182-e4c4-11ee-95c9-87774f69a715/ceph-volume.log.gz 2024-03-18T01:51:20.214 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:51:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:51:20.090+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:51:20.214 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:51:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:51:20.090+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:51:20.214 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:51:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:51:20.090+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:51:20.215 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:51:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:51:20.091+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:51:20.215 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:51:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:51:19.846+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:51:20.359 INFO:teuthology.orchestra.run.smithi029.stderr:gzip: /var/log/ceph/37f51182-e4c4-11ee-95c9-87774f69a715/ceph-mgr.smithi029.qviwhl.log: file size changed while zipping 2024-03-18T01:51:20.359 INFO:teuthology.orchestra.run.smithi029.stderr: 89.8% -- replaced with /var/log/ceph/37f51182-e4c4-11ee-95c9-87774f69a715/ceph-mgr.smithi029.qviwhl.log.gz 2024-03-18T01:51:20.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:51:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:51:20.218+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:51:20.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:51:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:51:20.218+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:51:20.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:51:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:51:20.218+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:51:20.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:51:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:51:20.218+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:51:20.803 INFO:teuthology.orchestra.run.smithi029.stderr:gzip: /var/log/ceph/37f51182-e4c4-11ee-95c9-87774f69a715/ceph-mon.smithi029.log: file size changed while zipping 2024-03-18T01:51:20.803 INFO:teuthology.orchestra.run.smithi029.stderr: 92.6% -- replaced with /var/log/ceph/37f51182-e4c4-11ee-95c9-87774f69a715/ceph-mon.smithi029.log.gz 2024-03-18T01:51:21.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:51:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:51:20.863+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:51:21.070 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:51:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:51:20.697+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 28488 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:51:21.114 INFO:teuthology.orchestra.run.smithi029.stderr: 95.5% -- replaced with /var/log/ceph/37f51182-e4c4-11ee-95c9-87774f69a715/ceph-mds.cephfs.smithi029.xuxvlr.log.gz 2024-03-18T01:51:21.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:51:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:51:21.211+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:51:21.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:51:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:51:21.211+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:51:21.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:51:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:51:21.211+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:51:21.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:51:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:51:21.211+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:51:21.571 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:51:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:51:21.132+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:51:21.571 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:51:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:51:21.132+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:51:21.571 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:51:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:51:21.132+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:51:21.571 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:51:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:51:21.133+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:51:21.631 INFO:teuthology.orchestra.run.smithi029.stderr:gzip: /var/log/ceph/37f51182-e4c4-11ee-95c9-87774f69a715/ceph-osd.2.log: file size changed while zipping 2024-03-18T01:51:21.682 INFO:teuthology.orchestra.run.smithi029.stderr: 91.0% -- replaced with /var/log/ceph/37f51182-e4c4-11ee-95c9-87774f69a715/ceph-osd.2.log.gz 2024-03-18T01:51:22.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:51:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:51:21.872+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:51:22.251 INFO:teuthology.orchestra.run.smithi029.stderr:gzip: /var/log/ceph/37f51182-e4c4-11ee-95c9-87774f69a715/ceph-osd.1.log: file size changed while zipping 2024-03-18T01:51:22.252 INFO:teuthology.orchestra.run.smithi029.stderr: 91.1% -- replaced with /var/log/ceph/37f51182-e4c4-11ee-95c9-87774f69a715/ceph-osd.1.log.gz 2024-03-18T01:51:22.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:51:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:51:22.198+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:51:22.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:51:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:51:22.198+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:51:22.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:51:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:51:22.198+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:51:22.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:51:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:51:22.199+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:51:22.570 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:51:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:51:22.121+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:51:22.571 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:51:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:51:22.121+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:51:22.571 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:51:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:51:22.121+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:51:22.571 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:51:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:51:22.122+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:51:22.654 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.90 2024-03-18T01:51:23.138 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:51:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:51:22.834+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:51:23.570 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:51:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:51:23.137+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:51:23.570 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:51:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:51:23.137+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:51:23.570 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:51:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:51:23.137+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:51:23.570 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:51:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:51:23.138+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:51:23.571 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:51:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:51:23.193+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:51:23.571 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:51:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:51:23.193+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:51:23.571 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:51:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:51:23.193+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:51:23.571 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:51:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:51:23.194+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:51:24.134 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:51:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:51:23.882+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:51:24.134 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:51:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:51:24.131+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:51:24.135 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:51:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:51:24.131+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:51:24.135 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:51:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:51:24.131+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:51:24.135 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:51:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:51:24.133+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:51:24.443 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:51:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:51:24.209+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:51:24.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:51:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:51:24.209+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:51:24.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:51:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:51:24.209+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:51:24.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:51:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:51:24.210+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:51:25.145 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:51:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:51:24.847+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:51:25.441 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:51:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:51:25.250+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:51:25.441 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:51:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:51:25.250+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:51:25.441 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:51:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:51:25.250+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:51:25.441 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:51:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:51:25.250+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:51:25.442 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:51:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:51:25.143+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:51:25.442 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:51:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:51:25.143+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:51:25.442 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:51:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:51:25.143+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:51:25.442 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:51:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:51:25.144+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:51:25.981 INFO:teuthology.orchestra.run.smithi029.stderr:gzip: /var/log/ceph/37f51182-e4c4-11ee-95c9-87774f69a715/ceph-osd.0.log: file size changed while zipping 2024-03-18T01:51:25.981 INFO:teuthology.orchestra.run.smithi029.stderr: 96.5% -- replaced with /var/log/ceph/37f51182-e4c4-11ee-95c9-87774f69a715/ceph-osd.0.log.gz 2024-03-18T01:51:26.069 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:51:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:51:25.700+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 28572 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:51:26.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:51:25 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:51:25.871+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:51:26.569 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:51:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:51:26.266+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:51:26.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:51:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:51:26.266+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:51:26.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:51:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:51:26.266+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:51:26.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:51:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:51:26.266+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:51:26.570 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:51:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:51:26.160+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:51:26.570 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:51:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:51:26.160+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:51:26.571 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:51:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:51:26.160+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:51:26.571 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:51:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:51:26.160+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:51:27.151 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:51:26 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:51:26.845+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:51:27.570 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:51:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:51:27.150+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:51:27.570 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:51:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:51:27.150+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:51:27.570 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:51:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:51:27.150+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:54.238243+0000 (oldest deadline 2024-03-18T01:41:14.238243+0000) 2024-03-18T01:51:27.570 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:51:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:51:27.151+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:51:27.571 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:51:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:51:27.277+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:51:27.571 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:51:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:51:27.277+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:51:27.571 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:51:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:51:27.277+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:40:53.827961+0000 (oldest deadline 2024-03-18T01:41:13.827961+0000) 2024-03-18T01:51:27.571 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:51:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:51:27.277+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:51:28.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:51:27 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:51:27.796+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:51:28.569 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:51:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:51:28.190+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:51:28.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:51:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:51:28.286+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:51:29.115 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:51:28 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:51:28.839+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:51:29.570 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:51:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:51:29.205+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:51:29.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:51:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:51:29.334+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:51:30.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:51:29 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:51:29.808+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:51:30.569 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:51:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:51:30.189+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:51:30.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:51:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:51:30.357+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:51:30.813 INFO:teuthology.orchestra.run.smithi029.stderr: 92.3% -- replaced with /var/log/ceph/37f51182-e4c4-11ee-95c9-87774f69a715/ceph-mds.cephfs.smithi029.brxqeh.log.gz 2024-03-18T01:51:30.815 INFO:teuthology.orchestra.run.smithi029.stderr: 2024-03-18T01:51:30.816 INFO:teuthology.orchestra.run.smithi029.stderr:real 0m11.206s 2024-03-18T01:51:30.816 INFO:teuthology.orchestra.run.smithi029.stderr:user 0m24.905s 2024-03-18T01:51:30.816 INFO:teuthology.orchestra.run.smithi029.stderr:sys 0m0.584s 2024-03-18T01:51:31.069 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:51:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:51:30.703+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 28655 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:51:31.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:51:30 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:51:30.770+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:51:31.569 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:51:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:51:31.196+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:51:31.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:51:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:51:31.308+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:51:31.655 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2024-03-18T01:51:31.656 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-18T01:51:32.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:51:31 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:51:31.812+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:51:32.569 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:51:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:51:32.234+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:51:32.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:51:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:51:32.326+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:51:33.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:51:32 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:51:32.771+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:51:33.569 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:51:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:51:33.326+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:51:33.570 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:51:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:51:33.243+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:51:34.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:51:33 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:51:33.762+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:51:34.569 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:51:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:51:34.373+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:51:34.570 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:51:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:51:34.245+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:51:34.718 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.90 2024-03-18T01:51:35.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:51:34 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:51:34.747+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:51:35.569 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:51:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:51:35.225+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:51:35.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:51:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:51:35.329+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:51:36.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:51:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:51:35.710+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:51:36.070 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:51:35 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:51:35.706+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 28738 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:51:36.569 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:51:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:51:36.221+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:51:36.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:51:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:51:36.313+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:51:37.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:51:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:51:36.721+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:51:15.812746+0000 (oldest deadline 2024-03-18T01:51:35.812746+0000) 2024-03-18T01:51:37.072 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:51:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:51:36.721+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:51:15.812746+0000 (oldest deadline 2024-03-18T01:51:35.812746+0000) 2024-03-18T01:51:37.072 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:51:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:51:36.721+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:51:15.812746+0000 (oldest deadline 2024-03-18T01:51:35.812746+0000) 2024-03-18T01:51:37.072 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:51:36 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:51:36.722+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:51:37.570 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:51:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:51:37.194+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:51:37.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:51:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:51:37.328+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:51:38.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:51:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:51:37.743+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:51:15.812746+0000 (oldest deadline 2024-03-18T01:51:35.812746+0000) 2024-03-18T01:51:38.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:51:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:51:37.743+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:51:15.812746+0000 (oldest deadline 2024-03-18T01:51:35.812746+0000) 2024-03-18T01:51:38.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:51:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:51:37.743+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:51:15.812746+0000 (oldest deadline 2024-03-18T01:51:35.812746+0000) 2024-03-18T01:51:38.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:51:37 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:51:37.744+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:51:38.569 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:51:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:51:38.182+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:51:38.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:51:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:51:38.295+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:51:39.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:51:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:51:38.698+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:51:15.812746+0000 (oldest deadline 2024-03-18T01:51:35.812746+0000) 2024-03-18T01:51:39.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:51:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:51:38.698+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:51:15.812746+0000 (oldest deadline 2024-03-18T01:51:35.812746+0000) 2024-03-18T01:51:39.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:51:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:51:38.698+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:51:15.812746+0000 (oldest deadline 2024-03-18T01:51:35.812746+0000) 2024-03-18T01:51:39.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:51:38 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:51:38.700+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:51:39.569 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:51:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:51:39.205+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:51:39.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:51:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:51:39.294+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:51:40.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:51:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:51:39.724+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:51:15.812746+0000 (oldest deadline 2024-03-18T01:51:35.812746+0000) 2024-03-18T01:51:40.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:51:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:51:39.724+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:51:15.812746+0000 (oldest deadline 2024-03-18T01:51:35.812746+0000) 2024-03-18T01:51:40.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:51:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:51:39.724+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:51:15.812746+0000 (oldest deadline 2024-03-18T01:51:35.812746+0000) 2024-03-18T01:51:40.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:51:39 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:51:39.725+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:51:40.569 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:51:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:51:40.206+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:51:40.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:51:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:51:40.250+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:51:41.069 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:51:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:51:40.709+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 28820 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:51:41.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:51:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:51:40.756+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:51:15.812746+0000 (oldest deadline 2024-03-18T01:51:35.812746+0000) 2024-03-18T01:51:41.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:51:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:51:40.756+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:51:15.812746+0000 (oldest deadline 2024-03-18T01:51:35.812746+0000) 2024-03-18T01:51:41.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:51:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:51:40.756+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:51:15.812746+0000 (oldest deadline 2024-03-18T01:51:35.812746+0000) 2024-03-18T01:51:41.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:51:40 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:51:40.756+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:51:41.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:51:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:51:41.276+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:51:41.570 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:51:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:51:41.179+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:51:42.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:51:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:51:41.806+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:51:15.812746+0000 (oldest deadline 2024-03-18T01:51:35.812746+0000) 2024-03-18T01:51:42.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:51:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:51:41.806+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:51:15.812746+0000 (oldest deadline 2024-03-18T01:51:35.812746+0000) 2024-03-18T01:51:42.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:51:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:51:41.806+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:51:15.812746+0000 (oldest deadline 2024-03-18T01:51:35.812746+0000) 2024-03-18T01:51:42.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:51:41 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:51:41.807+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:51:42.569 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:51:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:51:42.228+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:51:42.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:51:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:51:42.284+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:51:43.242 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:51:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:51:42.825+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:51:15.812746+0000 (oldest deadline 2024-03-18T01:51:35.812746+0000) 2024-03-18T01:51:43.242 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:51:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:51:42.825+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:51:15.812746+0000 (oldest deadline 2024-03-18T01:51:35.812746+0000) 2024-03-18T01:51:43.242 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:51:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:51:42.825+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:51:15.812746+0000 (oldest deadline 2024-03-18T01:51:35.812746+0000) 2024-03-18T01:51:43.242 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:51:42 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:51:42.826+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:51:43.569 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:51:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:51:43.279+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:51:43.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:51:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:51:43.240+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:51:44.263 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:51:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:51:43.875+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:51:15.812746+0000 (oldest deadline 2024-03-18T01:51:35.812746+0000) 2024-03-18T01:51:44.263 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:51:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:51:43.875+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:51:15.812746+0000 (oldest deadline 2024-03-18T01:51:35.812746+0000) 2024-03-18T01:51:44.263 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:51:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:51:43.875+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:51:15.812746+0000 (oldest deadline 2024-03-18T01:51:35.812746+0000) 2024-03-18T01:51:44.263 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:51:43 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:51:43.876+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:51:44.569 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:51:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:51:44.307+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:51:44.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:51:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:51:44.261+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:51:45.240 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:51:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:51:44.908+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:51:15.812746+0000 (oldest deadline 2024-03-18T01:51:35.812746+0000) 2024-03-18T01:51:45.240 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:51:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:51:44.908+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:51:15.812746+0000 (oldest deadline 2024-03-18T01:51:35.812746+0000) 2024-03-18T01:51:45.240 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:51:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:51:44.908+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:51:15.812746+0000 (oldest deadline 2024-03-18T01:51:35.812746+0000) 2024-03-18T01:51:45.241 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:51:44 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:51:44.908+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:51:45.570 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:51:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:51:45.307+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:51:45.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:51:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:51:45.238+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:51:46.069 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:51:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:51:45.712+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 28904 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:51:46.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:51:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:51:45.957+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:51:15.812746+0000 (oldest deadline 2024-03-18T01:51:35.812746+0000) 2024-03-18T01:51:46.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:51:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:51:45.957+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:51:15.812746+0000 (oldest deadline 2024-03-18T01:51:35.812746+0000) 2024-03-18T01:51:46.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:51:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:51:45.957+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:51:15.812746+0000 (oldest deadline 2024-03-18T01:51:35.812746+0000) 2024-03-18T01:51:46.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:51:45 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:51:45.958+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:51:46.569 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:51:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:51:46.310+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:51:46.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:51:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:51:46.255+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:51:46.720 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2024-03-18T01:51:46.720 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-18T01:51:47.261 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:51:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:51:46.995+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:51:15.812746+0000 (oldest deadline 2024-03-18T01:51:35.812746+0000) 2024-03-18T01:51:47.261 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:51:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:51:46.995+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:51:15.812746+0000 (oldest deadline 2024-03-18T01:51:35.812746+0000) 2024-03-18T01:51:47.261 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:51:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:51:46.995+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:51:15.812746+0000 (oldest deadline 2024-03-18T01:51:35.812746+0000) 2024-03-18T01:51:47.262 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:51:46 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:51:46.996+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:51:47.569 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:51:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:51:47.324+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:51:47.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:51:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:51:47.260+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:51:48.268 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:51:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:51:47.966+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:51:15.812746+0000 (oldest deadline 2024-03-18T01:51:35.812746+0000) 2024-03-18T01:51:48.268 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:51:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:51:47.966+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:51:15.812746+0000 (oldest deadline 2024-03-18T01:51:35.812746+0000) 2024-03-18T01:51:48.269 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:51:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:51:47.966+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:51:15.812746+0000 (oldest deadline 2024-03-18T01:51:35.812746+0000) 2024-03-18T01:51:48.269 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:51:47 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:51:47.966+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:51:48.570 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:51:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:51:48.366+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:51:48.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:51:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:51:48.267+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:51:28.001537+0000 (oldest deadline 2024-03-18T01:51:48.001537+0000) 2024-03-18T01:51:48.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:51:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:51:48.267+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:51:28.001537+0000 (oldest deadline 2024-03-18T01:51:48.001537+0000) 2024-03-18T01:51:48.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:51:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:51:48.267+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:51:28.001537+0000 (oldest deadline 2024-03-18T01:51:48.001537+0000) 2024-03-18T01:51:48.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:51:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:51:48.267+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:51:49.311 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:51:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:51:48.948+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:51:15.812746+0000 (oldest deadline 2024-03-18T01:51:35.812746+0000) 2024-03-18T01:51:49.311 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:51:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:51:48.948+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:51:15.812746+0000 (oldest deadline 2024-03-18T01:51:35.812746+0000) 2024-03-18T01:51:49.311 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:51:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:51:48.948+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:51:15.812746+0000 (oldest deadline 2024-03-18T01:51:35.812746+0000) 2024-03-18T01:51:49.311 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:51:48 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:51:48.948+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:51:49.570 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:51:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:51:49.386+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:51:28.702131+0000 (oldest deadline 2024-03-18T01:51:48.702131+0000) 2024-03-18T01:51:49.570 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:51:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:51:49.386+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:51:28.702131+0000 (oldest deadline 2024-03-18T01:51:48.702131+0000) 2024-03-18T01:51:49.570 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:51:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:51:49.387+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:51:28.702131+0000 (oldest deadline 2024-03-18T01:51:48.702131+0000) 2024-03-18T01:51:49.570 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:51:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:51:49.387+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:51:49.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:51:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:51:49.310+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:51:28.001537+0000 (oldest deadline 2024-03-18T01:51:48.001537+0000) 2024-03-18T01:51:49.571 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:51:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:51:49.310+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:51:28.001537+0000 (oldest deadline 2024-03-18T01:51:48.001537+0000) 2024-03-18T01:51:49.571 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:51:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:51:49.310+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:51:28.001537+0000 (oldest deadline 2024-03-18T01:51:48.001537+0000) 2024-03-18T01:51:49.571 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:51:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:51:49.310+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:51:49.794 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.90 2024-03-18T01:51:50.216 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:51:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:51:49.922+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:51:15.812746+0000 (oldest deadline 2024-03-18T01:51:35.812746+0000) 2024-03-18T01:51:50.216 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:51:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:51:49.922+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:51:15.812746+0000 (oldest deadline 2024-03-18T01:51:35.812746+0000) 2024-03-18T01:51:50.217 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:51:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:51:49.922+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:51:15.812746+0000 (oldest deadline 2024-03-18T01:51:35.812746+0000) 2024-03-18T01:51:50.217 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:51:49 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:51:49.923+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:51:50.569 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:51:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:51:50.434+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:51:28.702131+0000 (oldest deadline 2024-03-18T01:51:48.702131+0000) 2024-03-18T01:51:50.570 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:51:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:51:50.434+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:51:28.702131+0000 (oldest deadline 2024-03-18T01:51:48.702131+0000) 2024-03-18T01:51:50.570 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:51:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:51:50.434+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:51:28.702131+0000 (oldest deadline 2024-03-18T01:51:48.702131+0000) 2024-03-18T01:51:50.570 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:51:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:51:50.435+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:51:50.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:51:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:51:50.309+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:51:28.001537+0000 (oldest deadline 2024-03-18T01:51:48.001537+0000) 2024-03-18T01:51:50.571 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:51:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:51:50.309+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:51:28.001537+0000 (oldest deadline 2024-03-18T01:51:48.001537+0000) 2024-03-18T01:51:50.571 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:51:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:51:50.309+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:51:28.001537+0000 (oldest deadline 2024-03-18T01:51:48.001537+0000) 2024-03-18T01:51:50.571 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:51:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:51:50.309+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:51:51.069 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:51:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:51:50.716+0000 7f4cb4a92700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 28987 slow ops, oldest is log(1 entries from seq 283 at 2024-03-18T01:19:45.003350+0000) 2024-03-18T01:51:51.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:51:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:51:50.937+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:51:15.812746+0000 (oldest deadline 2024-03-18T01:51:35.812746+0000) 2024-03-18T01:51:51.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:51:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:51:50.937+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:51:15.812746+0000 (oldest deadline 2024-03-18T01:51:35.812746+0000) 2024-03-18T01:51:51.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:51:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:51:50.937+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:51:15.812746+0000 (oldest deadline 2024-03-18T01:51:35.812746+0000) 2024-03-18T01:51:51.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:51:50 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:51:50.938+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:51:51.569 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:51:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:51:51.271+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:51:28.001537+0000 (oldest deadline 2024-03-18T01:51:48.001537+0000) 2024-03-18T01:51:51.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:51:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:51:51.271+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:51:28.001537+0000 (oldest deadline 2024-03-18T01:51:48.001537+0000) 2024-03-18T01:51:51.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:51:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:51:51.271+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:51:28.001537+0000 (oldest deadline 2024-03-18T01:51:48.001537+0000) 2024-03-18T01:51:51.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:51:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:51:51.272+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:51:51.570 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:51:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:51:51.444+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:51:28.702131+0000 (oldest deadline 2024-03-18T01:51:48.702131+0000) 2024-03-18T01:51:51.570 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:51:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:51:51.444+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:51:28.702131+0000 (oldest deadline 2024-03-18T01:51:48.702131+0000) 2024-03-18T01:51:51.571 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:51:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:51:51.444+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:51:28.702131+0000 (oldest deadline 2024-03-18T01:51:48.702131+0000) 2024-03-18T01:51:51.571 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:51:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:51:51.445+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:51:52.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:51:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:51:51.920+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:51:15.812746+0000 (oldest deadline 2024-03-18T01:51:35.812746+0000) 2024-03-18T01:51:52.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:51:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:51:51.920+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:51:15.812746+0000 (oldest deadline 2024-03-18T01:51:35.812746+0000) 2024-03-18T01:51:52.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:51:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:51:51.920+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:51:15.812746+0000 (oldest deadline 2024-03-18T01:51:35.812746+0000) 2024-03-18T01:51:52.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:51:51 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:51:51.920+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:51:52.414 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:51:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:51:52.276+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:51:28.001537+0000 (oldest deadline 2024-03-18T01:51:48.001537+0000) 2024-03-18T01:51:52.414 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:51:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:51:52.276+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:51:28.001537+0000 (oldest deadline 2024-03-18T01:51:48.001537+0000) 2024-03-18T01:51:52.414 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:51:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:51:52.276+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:51:28.001537+0000 (oldest deadline 2024-03-18T01:51:48.001537+0000) 2024-03-18T01:51:52.414 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:51:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:51:52.277+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:51:52.795 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2024-03-18T01:51:52.795 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-18T01:51:52.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:51:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:51:52.412+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:51:28.702131+0000 (oldest deadline 2024-03-18T01:51:48.702131+0000) 2024-03-18T01:51:52.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:51:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:51:52.412+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:51:28.702131+0000 (oldest deadline 2024-03-18T01:51:48.702131+0000) 2024-03-18T01:51:52.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:51:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:51:52.412+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:51:28.702131+0000 (oldest deadline 2024-03-18T01:51:48.702131+0000) 2024-03-18T01:51:52.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:51:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:51:52.414+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:51:52.862 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.90 2024-03-18T01:51:52.862 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-03-18T01:51:52.863 INFO:tasks.cephadm:Removing cluster... 2024-03-18T01:51:52.863 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 37f51182-e4c4-11ee-95c9-87774f69a715 --force 2024-03-18T01:51:53.152 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:51:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:51:52.902+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:51:15.812746+0000 (oldest deadline 2024-03-18T01:51:35.812746+0000) 2024-03-18T01:51:53.152 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:51:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:51:52.902+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:51:15.812746+0000 (oldest deadline 2024-03-18T01:51:35.812746+0000) 2024-03-18T01:51:53.152 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:51:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:51:52.902+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:51:15.812746+0000 (oldest deadline 2024-03-18T01:51:35.812746+0000) 2024-03-18T01:51:53.153 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:51:52 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:51:52.904+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:51:53.409 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:51:53 smithi029 systemd[1]: Stopping Ceph mon.smithi029 for 37f51182-e4c4-11ee-95c9-87774f69a715... 2024-03-18T01:51:53.410 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:51:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:51:53.288+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:51:28.001537+0000 (oldest deadline 2024-03-18T01:51:48.001537+0000) 2024-03-18T01:51:53.410 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:51:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:51:53.288+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:51:28.001537+0000 (oldest deadline 2024-03-18T01:51:48.001537+0000) 2024-03-18T01:51:53.410 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:51:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:51:53.288+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:51:28.001537+0000 (oldest deadline 2024-03-18T01:51:48.001537+0000) 2024-03-18T01:51:53.410 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:51:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:51:53.288+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:51:53.702 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:51:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:51:53.417+0000 7f4cb7a98700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi029 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T01:51:53.703 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:51:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029[29327]: 2024-03-18T01:51:53.417+0000 7f4cb7a98700 -1 mon.smithi029@0(probing) e2 *** Got Signal Terminated *** 2024-03-18T01:51:53.703 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:51:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:51:53.408+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:51:28.702131+0000 (oldest deadline 2024-03-18T01:51:48.702131+0000) 2024-03-18T01:51:53.703 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:51:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:51:53.408+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:51:28.702131+0000 (oldest deadline 2024-03-18T01:51:48.702131+0000) 2024-03-18T01:51:53.703 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:51:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:51:53.408+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:51:28.702131+0000 (oldest deadline 2024-03-18T01:51:48.702131+0000) 2024-03-18T01:51:53.703 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:51:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:51:53.409+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:51:54.015 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:51:53 smithi029 podman[84294]: 2024-03-18 01:51:53.702262688 +0000 UTC m=+0.398069157 container died 0a7fff1d6dfcd675abb5a89188965e3ff6387fe9e1977d5eb7eafe0a990b5420 (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=quincy-6716c9f, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-03-18T01:51:54.015 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:51:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:51:53.940+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:51:15.812746+0000 (oldest deadline 2024-03-18T01:51:35.812746+0000) 2024-03-18T01:51:54.016 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:51:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:51:53.940+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:51:15.812746+0000 (oldest deadline 2024-03-18T01:51:35.812746+0000) 2024-03-18T01:51:54.016 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:51:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:51:53.940+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:51:15.812746+0000 (oldest deadline 2024-03-18T01:51:35.812746+0000) 2024-03-18T01:51:54.016 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:51:53 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:51:53.940+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:51:54.319 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:51:54 smithi029 podman[84294]: 2024-03-18 01:51:54.105957927 +0000 UTC m=+0.801764393 container cleanup 0a7fff1d6dfcd675abb5a89188965e3ff6387fe9e1977d5eb7eafe0a990b5420 (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=quincy-6716c9f, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-18T01:51:54.320 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:51:54 smithi029 bash[84294]: ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029 2024-03-18T01:51:54.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:51:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:51:54.253+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:51:28.001537+0000 (oldest deadline 2024-03-18T01:51:48.001537+0000) 2024-03-18T01:51:54.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:51:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:51:54.253+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:51:28.001537+0000 (oldest deadline 2024-03-18T01:51:48.001537+0000) 2024-03-18T01:51:54.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:51:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:51:54.253+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:51:28.001537+0000 (oldest deadline 2024-03-18T01:51:48.001537+0000) 2024-03-18T01:51:54.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:51:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:51:54.253+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:51:54.749 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:51:54 smithi029 podman[84308]: 2024-03-18 01:51:54.384591821 +0000 UTC m=+0.686662854 container remove 0a7fff1d6dfcd675abb5a89188965e3ff6387fe9e1977d5eb7eafe0a990b5420 (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-37f51182-e4c4-11ee-95c9-87774f69a715-mon-smithi029, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=quincy-6716c9f, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-18T01:51:54.749 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:51:54 smithi029 systemd[1]: ceph-37f51182-e4c4-11ee-95c9-87774f69a715@mon.smithi029.service: Deactivated successfully. 2024-03-18T01:51:54.750 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:51:54 smithi029 systemd[1]: Stopped Ceph mon.smithi029 for 37f51182-e4c4-11ee-95c9-87774f69a715. 2024-03-18T01:51:54.750 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Mar 18 01:51:54 smithi029 systemd[1]: ceph-37f51182-e4c4-11ee-95c9-87774f69a715@mon.smithi029.service: Consumed 21.026s CPU time. 2024-03-18T01:51:54.750 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:51:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:51:54.444+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:51:28.702131+0000 (oldest deadline 2024-03-18T01:51:48.702131+0000) 2024-03-18T01:51:54.750 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:51:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:51:54.444+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:51:28.702131+0000 (oldest deadline 2024-03-18T01:51:48.702131+0000) 2024-03-18T01:51:54.750 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:51:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:51:54.444+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:51:28.702131+0000 (oldest deadline 2024-03-18T01:51:48.702131+0000) 2024-03-18T01:51:54.750 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:51:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:51:54.445+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:51:55.033 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:51:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:51:54.923+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:51:15.812746+0000 (oldest deadline 2024-03-18T01:51:35.812746+0000) 2024-03-18T01:51:55.033 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:51:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:51:54.923+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:51:15.812746+0000 (oldest deadline 2024-03-18T01:51:35.812746+0000) 2024-03-18T01:51:55.033 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:51:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:51:54.923+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:51:15.812746+0000 (oldest deadline 2024-03-18T01:51:35.812746+0000) 2024-03-18T01:51:55.033 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:51:54 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:51:54.924+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:51:55.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:51:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:51:55.228+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:51:28.001537+0000 (oldest deadline 2024-03-18T01:51:48.001537+0000) 2024-03-18T01:51:55.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:51:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:51:55.228+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:51:28.001537+0000 (oldest deadline 2024-03-18T01:51:48.001537+0000) 2024-03-18T01:51:55.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:51:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:51:55.228+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:51:28.001537+0000 (oldest deadline 2024-03-18T01:51:48.001537+0000) 2024-03-18T01:51:55.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:51:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:51:55.228+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:51:55.741 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:51:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:51:55.468+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:51:28.702131+0000 (oldest deadline 2024-03-18T01:51:48.702131+0000) 2024-03-18T01:51:55.741 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:51:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:51:55.468+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:51:28.702131+0000 (oldest deadline 2024-03-18T01:51:48.702131+0000) 2024-03-18T01:51:55.742 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:51:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:51:55.468+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:51:28.702131+0000 (oldest deadline 2024-03-18T01:51:48.702131+0000) 2024-03-18T01:51:55.742 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:51:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:51:55.469+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:51:55.997 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:51:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:51:55.967+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:51:15.812746+0000 (oldest deadline 2024-03-18T01:51:35.812746+0000) 2024-03-18T01:51:55.997 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:51:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:51:55.967+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:51:15.812746+0000 (oldest deadline 2024-03-18T01:51:35.812746+0000) 2024-03-18T01:51:55.997 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:51:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:51:55.967+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:51:15.812746+0000 (oldest deadline 2024-03-18T01:51:35.812746+0000) 2024-03-18T01:51:55.997 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:51:55 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:51:55.968+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:51:56.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:51:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:51:56.196+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:51:28.001537+0000 (oldest deadline 2024-03-18T01:51:48.001537+0000) 2024-03-18T01:51:56.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:51:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:51:56.196+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:51:28.001537+0000 (oldest deadline 2024-03-18T01:51:48.001537+0000) 2024-03-18T01:51:56.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:51:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:51:56.196+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:51:28.001537+0000 (oldest deadline 2024-03-18T01:51:48.001537+0000) 2024-03-18T01:51:56.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:51:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:51:56.196+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:51:56.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:51:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:51:56.444+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:51:28.702131+0000 (oldest deadline 2024-03-18T01:51:48.702131+0000) 2024-03-18T01:51:56.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:51:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:51:56.444+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:51:28.702131+0000 (oldest deadline 2024-03-18T01:51:48.702131+0000) 2024-03-18T01:51:56.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:51:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:51:56.444+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:51:28.702131+0000 (oldest deadline 2024-03-18T01:51:48.702131+0000) 2024-03-18T01:51:56.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:51:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:51:56.445+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:51:57.203 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:51:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:51:56.958+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:51:15.812746+0000 (oldest deadline 2024-03-18T01:51:35.812746+0000) 2024-03-18T01:51:57.203 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:51:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:51:56.958+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:51:15.812746+0000 (oldest deadline 2024-03-18T01:51:35.812746+0000) 2024-03-18T01:51:57.203 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:51:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:51:56.958+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:51:15.812746+0000 (oldest deadline 2024-03-18T01:51:35.812746+0000) 2024-03-18T01:51:57.203 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:51:56 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:51:56.959+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:51:57.454 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:51:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:51:57.202+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:51:28.001537+0000 (oldest deadline 2024-03-18T01:51:48.001537+0000) 2024-03-18T01:51:57.454 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:51:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:51:57.202+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:51:28.001537+0000 (oldest deadline 2024-03-18T01:51:48.001537+0000) 2024-03-18T01:51:57.455 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:51:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:51:57.202+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:51:28.001537+0000 (oldest deadline 2024-03-18T01:51:48.001537+0000) 2024-03-18T01:51:57.455 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:51:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:51:57.202+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:51:57.455 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:51:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:51:57.447+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:51:28.702131+0000 (oldest deadline 2024-03-18T01:51:48.702131+0000) 2024-03-18T01:51:57.455 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:51:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:51:57.447+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:51:28.702131+0000 (oldest deadline 2024-03-18T01:51:48.702131+0000) 2024-03-18T01:51:57.455 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:51:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:51:57.447+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:51:28.702131+0000 (oldest deadline 2024-03-18T01:51:48.702131+0000) 2024-03-18T01:51:57.456 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:51:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:51:57.448+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:51:58.250 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:51:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:51:57.972+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:51:15.812746+0000 (oldest deadline 2024-03-18T01:51:35.812746+0000) 2024-03-18T01:51:58.250 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:51:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:51:57.972+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:51:15.812746+0000 (oldest deadline 2024-03-18T01:51:35.812746+0000) 2024-03-18T01:51:58.251 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:51:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:51:57.972+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:51:15.812746+0000 (oldest deadline 2024-03-18T01:51:35.812746+0000) 2024-03-18T01:51:58.251 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:51:57 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:51:57.973+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:51:58.570 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:51:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:51:58.417+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:51:28.702131+0000 (oldest deadline 2024-03-18T01:51:48.702131+0000) 2024-03-18T01:51:58.570 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:51:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:51:58.417+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:51:28.702131+0000 (oldest deadline 2024-03-18T01:51:48.702131+0000) 2024-03-18T01:51:58.570 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:51:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:51:58.417+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:51:28.702131+0000 (oldest deadline 2024-03-18T01:51:48.702131+0000) 2024-03-18T01:51:58.570 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:51:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:51:58.417+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:51:58.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:51:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:51:58.249+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:51:28.001537+0000 (oldest deadline 2024-03-18T01:51:48.001537+0000) 2024-03-18T01:51:58.571 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:51:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:51:58.249+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:51:28.001537+0000 (oldest deadline 2024-03-18T01:51:48.001537+0000) 2024-03-18T01:51:58.571 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:51:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:51:58.249+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:51:28.001537+0000 (oldest deadline 2024-03-18T01:51:48.001537+0000) 2024-03-18T01:51:58.571 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:51:58 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:51:58.249+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:51:59.224 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:51:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:51:59.005+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:51:15.812746+0000 (oldest deadline 2024-03-18T01:51:35.812746+0000) 2024-03-18T01:51:59.224 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:51:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:51:59.005+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:51:15.812746+0000 (oldest deadline 2024-03-18T01:51:35.812746+0000) 2024-03-18T01:51:59.224 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:51:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:51:59.005+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:51:15.812746+0000 (oldest deadline 2024-03-18T01:51:35.812746+0000) 2024-03-18T01:51:59.224 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:51:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:51:59.006+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:51:59.569 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:51:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:51:59.380+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:51:28.702131+0000 (oldest deadline 2024-03-18T01:51:48.702131+0000) 2024-03-18T01:51:59.570 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:51:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:51:59.380+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:51:28.702131+0000 (oldest deadline 2024-03-18T01:51:48.702131+0000) 2024-03-18T01:51:59.570 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:51:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:51:59.380+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:51:28.702131+0000 (oldest deadline 2024-03-18T01:51:48.702131+0000) 2024-03-18T01:51:59.570 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:51:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:51:59.380+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:51:59.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:51:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:51:59.222+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:51:28.001537+0000 (oldest deadline 2024-03-18T01:51:48.001537+0000) 2024-03-18T01:51:59.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:51:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:51:59.222+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:51:28.001537+0000 (oldest deadline 2024-03-18T01:51:48.001537+0000) 2024-03-18T01:51:59.571 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:51:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:51:59.222+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:51:28.001537+0000 (oldest deadline 2024-03-18T01:51:48.001537+0000) 2024-03-18T01:51:59.571 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:51:59 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:51:59.223+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:52:00.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:52:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:52:00.016+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:51:15.812746+0000 (oldest deadline 2024-03-18T01:51:35.812746+0000) 2024-03-18T01:52:00.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:52:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:52:00.016+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:51:15.812746+0000 (oldest deadline 2024-03-18T01:51:35.812746+0000) 2024-03-18T01:52:00.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:52:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:52:00.016+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:51:15.812746+0000 (oldest deadline 2024-03-18T01:51:35.812746+0000) 2024-03-18T01:52:00.228 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:52:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:52:00.016+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:52:00.569 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:52:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:52:00.400+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:51:28.702131+0000 (oldest deadline 2024-03-18T01:51:48.702131+0000) 2024-03-18T01:52:00.570 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:52:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:52:00.400+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:51:28.702131+0000 (oldest deadline 2024-03-18T01:51:48.702131+0000) 2024-03-18T01:52:00.570 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:52:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:52:00.400+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:51:28.702131+0000 (oldest deadline 2024-03-18T01:51:48.702131+0000) 2024-03-18T01:52:00.570 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:52:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:52:00.401+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:52:00.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:52:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:52:00.226+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:51:28.001537+0000 (oldest deadline 2024-03-18T01:51:48.001537+0000) 2024-03-18T01:52:00.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:52:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:52:00.226+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:51:28.001537+0000 (oldest deadline 2024-03-18T01:51:48.001537+0000) 2024-03-18T01:52:00.571 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:52:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:52:00.226+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:51:28.001537+0000 (oldest deadline 2024-03-18T01:51:48.001537+0000) 2024-03-18T01:52:00.571 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:52:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:52:00.226+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:52:01.027 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:52:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:52:00.981+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:51:15.812746+0000 (oldest deadline 2024-03-18T01:51:35.812746+0000) 2024-03-18T01:52:01.028 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:52:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:52:00.981+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:51:15.812746+0000 (oldest deadline 2024-03-18T01:51:35.812746+0000) 2024-03-18T01:52:01.028 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:52:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:52:00.981+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:51:15.812746+0000 (oldest deadline 2024-03-18T01:51:35.812746+0000) 2024-03-18T01:52:01.028 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:52:00 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:52:00.981+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:52:01.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:52:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:52:01.248+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:51:28.001537+0000 (oldest deadline 2024-03-18T01:51:48.001537+0000) 2024-03-18T01:52:01.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:52:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:52:01.248+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:51:28.001537+0000 (oldest deadline 2024-03-18T01:51:48.001537+0000) 2024-03-18T01:52:01.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:52:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:52:01.248+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:51:28.001537+0000 (oldest deadline 2024-03-18T01:51:48.001537+0000) 2024-03-18T01:52:01.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:52:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:52:01.249+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:52:01.756 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:52:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:52:01.434+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:51:28.702131+0000 (oldest deadline 2024-03-18T01:51:48.702131+0000) 2024-03-18T01:52:01.756 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:52:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:52:01.434+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:51:28.702131+0000 (oldest deadline 2024-03-18T01:51:48.702131+0000) 2024-03-18T01:52:01.757 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:52:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:52:01.434+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:51:28.702131+0000 (oldest deadline 2024-03-18T01:51:48.702131+0000) 2024-03-18T01:52:01.757 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:52:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:52:01.434+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:52:02.035 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:52:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:52:01.960+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:51:15.812746+0000 (oldest deadline 2024-03-18T01:51:35.812746+0000) 2024-03-18T01:52:02.035 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:52:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:52:01.960+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:51:15.812746+0000 (oldest deadline 2024-03-18T01:51:35.812746+0000) 2024-03-18T01:52:02.036 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:52:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:52:01.960+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:51:15.812746+0000 (oldest deadline 2024-03-18T01:51:35.812746+0000) 2024-03-18T01:52:02.036 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:52:01 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:52:01.961+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:52:02.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:52:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:52:02.243+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:51:28.001537+0000 (oldest deadline 2024-03-18T01:51:48.001537+0000) 2024-03-18T01:52:02.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:52:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:52:02.243+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:51:28.001537+0000 (oldest deadline 2024-03-18T01:51:48.001537+0000) 2024-03-18T01:52:02.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:52:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:52:02.243+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:51:28.001537+0000 (oldest deadline 2024-03-18T01:51:48.001537+0000) 2024-03-18T01:52:02.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:52:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:52:02.243+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:52:02.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:52:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:52:02.385+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:51:28.702131+0000 (oldest deadline 2024-03-18T01:51:48.702131+0000) 2024-03-18T01:52:02.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:52:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:52:02.385+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:51:28.702131+0000 (oldest deadline 2024-03-18T01:51:48.702131+0000) 2024-03-18T01:52:02.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:52:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:52:02.385+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:51:28.702131+0000 (oldest deadline 2024-03-18T01:51:48.702131+0000) 2024-03-18T01:52:02.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:52:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:52:02.386+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:52:02.984 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:52:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:52:02.926+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:51:15.812746+0000 (oldest deadline 2024-03-18T01:51:35.812746+0000) 2024-03-18T01:52:02.984 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:52:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:52:02.926+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:51:15.812746+0000 (oldest deadline 2024-03-18T01:51:35.812746+0000) 2024-03-18T01:52:02.984 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:52:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:52:02.926+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:51:15.812746+0000 (oldest deadline 2024-03-18T01:51:35.812746+0000) 2024-03-18T01:52:02.984 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:52:02 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:52:02.927+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:52:03.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:52:03 smithi029 systemd[1]: Stopping Ceph osd.0 for 37f51182-e4c4-11ee-95c9-87774f69a715... 2024-03-18T01:52:03.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:52:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:52:03.228+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:51:28.001537+0000 (oldest deadline 2024-03-18T01:51:48.001537+0000) 2024-03-18T01:52:03.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:52:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:52:03.228+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:51:28.001537+0000 (oldest deadline 2024-03-18T01:51:48.001537+0000) 2024-03-18T01:52:03.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:52:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:52:03.228+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:51:28.001537+0000 (oldest deadline 2024-03-18T01:51:48.001537+0000) 2024-03-18T01:52:03.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:52:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:52:03.229+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:52:03.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:52:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:52:03.340+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:51:28.702131+0000 (oldest deadline 2024-03-18T01:51:48.702131+0000) 2024-03-18T01:52:03.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:52:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:52:03.340+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:51:28.702131+0000 (oldest deadline 2024-03-18T01:51:48.702131+0000) 2024-03-18T01:52:03.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:52:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:52:03.340+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:51:28.702131+0000 (oldest deadline 2024-03-18T01:51:48.702131+0000) 2024-03-18T01:52:03.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:52:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:52:03.340+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:52:03.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:52:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:52:03.429+0000 7f7851fed700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T01:52:03.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:52:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:52:03.429+0000 7f7851fed700 -1 osd.0 46 *** Got signal Terminated *** 2024-03-18T01:52:03.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:52:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:52:03.429+0000 7f7851fed700 -1 osd.0 46 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-18T01:52:04.232 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:52:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:52:03.926+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:51:15.812746+0000 (oldest deadline 2024-03-18T01:51:35.812746+0000) 2024-03-18T01:52:04.232 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:52:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:52:03.926+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:51:15.812746+0000 (oldest deadline 2024-03-18T01:51:35.812746+0000) 2024-03-18T01:52:04.232 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:52:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:52:03.926+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:51:15.812746+0000 (oldest deadline 2024-03-18T01:51:35.812746+0000) 2024-03-18T01:52:04.232 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:52:03 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:52:03.926+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:52:04.569 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:52:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:52:04.296+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:51:28.702131+0000 (oldest deadline 2024-03-18T01:51:48.702131+0000) 2024-03-18T01:52:04.570 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:52:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:52:04.296+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:51:28.702131+0000 (oldest deadline 2024-03-18T01:51:48.702131+0000) 2024-03-18T01:52:04.570 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:52:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:52:04.296+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:51:28.702131+0000 (oldest deadline 2024-03-18T01:51:48.702131+0000) 2024-03-18T01:52:04.570 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:52:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:52:04.296+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:52:04.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:52:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:52:04.231+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:51:28.001537+0000 (oldest deadline 2024-03-18T01:51:48.001537+0000) 2024-03-18T01:52:04.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:52:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:52:04.231+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:51:28.001537+0000 (oldest deadline 2024-03-18T01:51:48.001537+0000) 2024-03-18T01:52:04.571 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:52:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:52:04.231+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:51:28.001537+0000 (oldest deadline 2024-03-18T01:51:48.001537+0000) 2024-03-18T01:52:04.571 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:52:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:52:04.231+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:52:05.229 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:52:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:52:04.884+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:51:15.812746+0000 (oldest deadline 2024-03-18T01:51:35.812746+0000) 2024-03-18T01:52:05.229 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:52:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:52:04.884+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:51:15.812746+0000 (oldest deadline 2024-03-18T01:51:35.812746+0000) 2024-03-18T01:52:05.229 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:52:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:52:04.884+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:51:15.812746+0000 (oldest deadline 2024-03-18T01:51:35.812746+0000) 2024-03-18T01:52:05.229 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:52:04 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:52:04.885+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:52:05.569 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:52:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:52:05.264+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:51:28.702131+0000 (oldest deadline 2024-03-18T01:51:48.702131+0000) 2024-03-18T01:52:05.570 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:52:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:52:05.264+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:51:28.702131+0000 (oldest deadline 2024-03-18T01:51:48.702131+0000) 2024-03-18T01:52:05.570 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:52:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:52:05.264+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:51:28.702131+0000 (oldest deadline 2024-03-18T01:51:48.702131+0000) 2024-03-18T01:52:05.570 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:52:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:52:05.266+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:52:05.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:52:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:52:05.227+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:51:28.001537+0000 (oldest deadline 2024-03-18T01:51:48.001537+0000) 2024-03-18T01:52:05.571 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:52:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:52:05.227+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:51:28.001537+0000 (oldest deadline 2024-03-18T01:51:48.001537+0000) 2024-03-18T01:52:05.571 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:52:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:52:05.227+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:51:28.001537+0000 (oldest deadline 2024-03-18T01:51:48.001537+0000) 2024-03-18T01:52:05.571 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:52:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:52:05.227+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:52:06.203 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:52:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:52:05.855+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:51:15.812746+0000 (oldest deadline 2024-03-18T01:51:35.812746+0000) 2024-03-18T01:52:06.203 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:52:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:52:05.855+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:51:15.812746+0000 (oldest deadline 2024-03-18T01:51:35.812746+0000) 2024-03-18T01:52:06.203 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:52:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:52:05.855+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:51:15.812746+0000 (oldest deadline 2024-03-18T01:51:35.812746+0000) 2024-03-18T01:52:06.203 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:52:05 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:52:05.855+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:52:06.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:52:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:52:06.201+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:51:28.001537+0000 (oldest deadline 2024-03-18T01:51:48.001537+0000) 2024-03-18T01:52:06.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:52:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:52:06.201+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:51:28.001537+0000 (oldest deadline 2024-03-18T01:51:48.001537+0000) 2024-03-18T01:52:06.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:52:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:52:06.201+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:51:28.001537+0000 (oldest deadline 2024-03-18T01:51:48.001537+0000) 2024-03-18T01:52:06.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:52:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:52:06.202+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:52:06.571 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:52:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:52:06.237+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:51:28.702131+0000 (oldest deadline 2024-03-18T01:51:48.702131+0000) 2024-03-18T01:52:06.571 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:52:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:52:06.237+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:51:28.702131+0000 (oldest deadline 2024-03-18T01:51:48.702131+0000) 2024-03-18T01:52:06.571 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:52:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:52:06.237+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:51:28.702131+0000 (oldest deadline 2024-03-18T01:51:48.702131+0000) 2024-03-18T01:52:06.571 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:52:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:52:06.238+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:52:07.180 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:52:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:52:06.839+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:51:15.812746+0000 (oldest deadline 2024-03-18T01:51:35.812746+0000) 2024-03-18T01:52:07.181 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:52:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:52:06.839+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:51:15.812746+0000 (oldest deadline 2024-03-18T01:51:35.812746+0000) 2024-03-18T01:52:07.181 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:52:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:52:06.839+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:51:15.812746+0000 (oldest deadline 2024-03-18T01:51:35.812746+0000) 2024-03-18T01:52:07.181 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:52:06 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:52:06.840+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:52:07.570 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:52:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:52:07.267+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:51:28.702131+0000 (oldest deadline 2024-03-18T01:51:48.702131+0000) 2024-03-18T01:52:07.570 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:52:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:52:07.267+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:51:28.702131+0000 (oldest deadline 2024-03-18T01:51:48.702131+0000) 2024-03-18T01:52:07.570 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:52:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:52:07.267+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:51:28.702131+0000 (oldest deadline 2024-03-18T01:51:48.702131+0000) 2024-03-18T01:52:07.570 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:52:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:52:07.268+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:52:07.571 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:52:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:52:07.179+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:51:28.001537+0000 (oldest deadline 2024-03-18T01:51:48.001537+0000) 2024-03-18T01:52:07.571 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:52:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:52:07.179+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:51:28.001537+0000 (oldest deadline 2024-03-18T01:51:48.001537+0000) 2024-03-18T01:52:07.571 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:52:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:52:07.179+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:51:28.001537+0000 (oldest deadline 2024-03-18T01:51:48.001537+0000) 2024-03-18T01:52:07.571 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:52:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:52:07.180+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:52:08.200 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:52:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:52:07.840+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:51:15.812746+0000 (oldest deadline 2024-03-18T01:51:35.812746+0000) 2024-03-18T01:52:08.200 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:52:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:52:07.840+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:51:15.812746+0000 (oldest deadline 2024-03-18T01:51:35.812746+0000) 2024-03-18T01:52:08.201 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:52:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:52:07.840+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:51:15.812746+0000 (oldest deadline 2024-03-18T01:51:35.812746+0000) 2024-03-18T01:52:08.201 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:52:07 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:52:07.841+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:52:08.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:52:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:52:08.198+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:51:28.001537+0000 (oldest deadline 2024-03-18T01:51:48.001537+0000) 2024-03-18T01:52:08.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:52:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:52:08.198+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:51:28.001537+0000 (oldest deadline 2024-03-18T01:51:48.001537+0000) 2024-03-18T01:52:08.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:52:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:52:08.198+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:51:28.001537+0000 (oldest deadline 2024-03-18T01:51:48.001537+0000) 2024-03-18T01:52:08.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:52:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:52:08.199+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:52:08.571 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:52:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:52:08.263+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:51:28.702131+0000 (oldest deadline 2024-03-18T01:51:48.702131+0000) 2024-03-18T01:52:08.571 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:52:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:52:08.263+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:51:28.702131+0000 (oldest deadline 2024-03-18T01:51:48.702131+0000) 2024-03-18T01:52:08.571 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:52:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:52:08.263+0000 7f784ddea700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:51:28.702131+0000 (oldest deadline 2024-03-18T01:51:48.702131+0000) 2024-03-18T01:52:08.571 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:52:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0[47706]: 2024-03-18T01:52:08.263+0000 7f784ddea700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.15:259 2.13 2:ceffbd45:::200.0000000a:head [write 1671927~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:52:09.249 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:52:09 smithi029 podman[85137]: 2024-03-18 01:52:09.066573508 +0000 UTC m=+5.748436108 container died b34def6ff829f3059d1c492527c0d9d055c38d08b00ec05783150222ea47dd3c (image=quay.ceph.io/ceph-ci/ceph@sha256:a75c953aa7dcb8597f235a84417a8be589209dba39db0305cebbb10d212afaf9, name=ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0, RELEASE=quincy-6716c9f, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-03-18T01:52:09.250 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:52:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:52:08.819+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:51:15.812746+0000 (oldest deadline 2024-03-18T01:51:35.812746+0000) 2024-03-18T01:52:09.250 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:52:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:52:08.819+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:51:15.812746+0000 (oldest deadline 2024-03-18T01:51:35.812746+0000) 2024-03-18T01:52:09.250 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:52:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:52:08.819+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:51:15.812746+0000 (oldest deadline 2024-03-18T01:51:35.812746+0000) 2024-03-18T01:52:09.250 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:52:08 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:52:08.820+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:52:09.570 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:52:09 smithi029 podman[85137]: 2024-03-18 01:52:09.409185976 +0000 UTC m=+6.091048582 container cleanup b34def6ff829f3059d1c492527c0d9d055c38d08b00ec05783150222ea47dd3c (image=quay.ceph.io/ceph-ci/ceph@sha256:a75c953aa7dcb8597f235a84417a8be589209dba39db0305cebbb10d212afaf9, name=ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, RELEASE=quincy-6716c9f, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-03-18T01:52:09.570 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:52:09 smithi029 bash[85137]: ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0 2024-03-18T01:52:09.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:52:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:52:09.248+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:51:28.001537+0000 (oldest deadline 2024-03-18T01:51:48.001537+0000) 2024-03-18T01:52:09.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:52:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:52:09.248+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:51:28.001537+0000 (oldest deadline 2024-03-18T01:51:48.001537+0000) 2024-03-18T01:52:09.571 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:52:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:52:09.248+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:51:28.001537+0000 (oldest deadline 2024-03-18T01:51:48.001537+0000) 2024-03-18T01:52:09.571 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:52:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:52:09.248+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:52:10.068 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:52:09 smithi029 podman[85157]: 2024-03-18 01:52:09.720121627 +0000 UTC m=+0.657663562 container remove b34def6ff829f3059d1c492527c0d9d055c38d08b00ec05783150222ea47dd3c (image=quay.ceph.io/ceph-ci/ceph@sha256:a75c953aa7dcb8597f235a84417a8be589209dba39db0305cebbb10d212afaf9, name=ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , RELEASE=quincy-6716c9f, org.label-schema.license=GPLv2) 2024-03-18T01:52:10.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:52:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:52:09.801+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:51:15.812746+0000 (oldest deadline 2024-03-18T01:51:35.812746+0000) 2024-03-18T01:52:10.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:52:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:52:09.801+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:51:15.812746+0000 (oldest deadline 2024-03-18T01:51:35.812746+0000) 2024-03-18T01:52:10.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:52:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:52:09.801+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:51:15.812746+0000 (oldest deadline 2024-03-18T01:51:35.812746+0000) 2024-03-18T01:52:10.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:52:09 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:52:09.801+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:52:10.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:52:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:52:10.271+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:51:28.001537+0000 (oldest deadline 2024-03-18T01:51:48.001537+0000) 2024-03-18T01:52:10.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:52:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:52:10.271+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:51:28.001537+0000 (oldest deadline 2024-03-18T01:51:48.001537+0000) 2024-03-18T01:52:10.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:52:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:52:10.271+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:51:28.001537+0000 (oldest deadline 2024-03-18T01:51:48.001537+0000) 2024-03-18T01:52:10.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:52:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:52:10.272+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:52:10.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:52:10 smithi029 podman[85252]: 2024-03-18 01:52:09.970697297 +0000 UTC m=+0.016826300 image pull 19857db3290c58169e44a8aec8ef9646981a77ffd5b146ecef1b6faf59d6c3ab quay.ceph.io/ceph-ci/ceph@sha256:a75c953aa7dcb8597f235a84417a8be589209dba39db0305cebbb10d212afaf9 2024-03-18T01:52:10.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:52:10 smithi029 podman[85252]: 2024-03-18 01:52:10.096875716 +0000 UTC m=+0.143004742 container create af352c684f2eb6f52f3d978937bb2314a7aafbc6e8bdf074149f7fe4b036477f (image=quay.ceph.io/ceph-ci/ceph@sha256:a75c953aa7dcb8597f235a84417a8be589209dba39db0305cebbb10d212afaf9, name=ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0-deactivate, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=quincy-6716c9f, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux ) 2024-03-18T01:52:10.597 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:52:10 smithi029 podman[85252]: 2024-03-18 01:52:10.34426306 +0000 UTC m=+0.390392060 container init af352c684f2eb6f52f3d978937bb2314a7aafbc6e8bdf074149f7fe4b036477f (image=quay.ceph.io/ceph-ci/ceph@sha256:a75c953aa7dcb8597f235a84417a8be589209dba39db0305cebbb10d212afaf9, name=ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0-deactivate, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.build-date=20240307, RELEASE=quincy-6716c9f, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-03-18T01:52:10.597 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:52:10 smithi029 podman[85252]: 2024-03-18 01:52:10.347544699 +0000 UTC m=+0.393673687 container start af352c684f2eb6f52f3d978937bb2314a7aafbc6e8bdf074149f7fe4b036477f (image=quay.ceph.io/ceph-ci/ceph@sha256:a75c953aa7dcb8597f235a84417a8be589209dba39db0305cebbb10d212afaf9, name=ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0-deactivate, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=quincy-6716c9f, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux ) 2024-03-18T01:52:10.597 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:52:10 smithi029 podman[85252]: 2024-03-18 01:52:10.453109468 +0000 UTC m=+0.499238471 container attach af352c684f2eb6f52f3d978937bb2314a7aafbc6e8bdf074149f7fe4b036477f (image=quay.ceph.io/ceph-ci/ceph@sha256:a75c953aa7dcb8597f235a84417a8be589209dba39db0305cebbb10d212afaf9, name=ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0-deactivate, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=quincy-6716c9f, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, ceph=True) 2024-03-18T01:52:10.967 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:52:10 smithi029 podman[85252]: 2024-03-18 01:52:10.645531217 +0000 UTC m=+0.691660205 container died af352c684f2eb6f52f3d978937bb2314a7aafbc6e8bdf074149f7fe4b036477f (image=quay.ceph.io/ceph-ci/ceph@sha256:a75c953aa7dcb8597f235a84417a8be589209dba39db0305cebbb10d212afaf9, name=ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0-deactivate, GIT_CLEAN=True, io.buildah.version=1.33.5, RELEASE=quincy-6716c9f, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS) 2024-03-18T01:52:10.967 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:52:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:52:10.829+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:51:15.812746+0000 (oldest deadline 2024-03-18T01:51:35.812746+0000) 2024-03-18T01:52:10.967 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:52:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:52:10.829+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:51:15.812746+0000 (oldest deadline 2024-03-18T01:51:35.812746+0000) 2024-03-18T01:52:10.968 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:52:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:52:10.829+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:51:15.812746+0000 (oldest deadline 2024-03-18T01:51:35.812746+0000) 2024-03-18T01:52:10.968 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:52:10 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:52:10.830+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:52:11.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:52:11 smithi029 systemd[1]: Stopping Ceph osd.1 for 37f51182-e4c4-11ee-95c9-87774f69a715... 2024-03-18T01:52:11.570 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:52:11 smithi029 podman[85279]: 2024-03-18 01:52:11.270075088 +0000 UTC m=+0.616324444 container remove af352c684f2eb6f52f3d978937bb2314a7aafbc6e8bdf074149f7fe4b036477f (image=quay.ceph.io/ceph-ci/ceph@sha256:a75c953aa7dcb8597f235a84417a8be589209dba39db0305cebbb10d212afaf9, name=ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-0-deactivate, RELEASE=quincy-6716c9f, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-18T01:52:11.570 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:52:11 smithi029 systemd[1]: ceph-37f51182-e4c4-11ee-95c9-87774f69a715@osd.0.service: Deactivated successfully. 2024-03-18T01:52:11.570 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:52:11 smithi029 systemd[1]: Stopped Ceph osd.0 for 37f51182-e4c4-11ee-95c9-87774f69a715. 2024-03-18T01:52:11.570 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 18 01:52:11 smithi029 systemd[1]: ceph-37f51182-e4c4-11ee-95c9-87774f69a715@osd.0.service: Consumed 30min 47.143s CPU time. 2024-03-18T01:52:11.571 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:52:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:52:11.282+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:51:28.001537+0000 (oldest deadline 2024-03-18T01:51:48.001537+0000) 2024-03-18T01:52:11.571 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:52:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:52:11.282+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:51:28.001537+0000 (oldest deadline 2024-03-18T01:51:48.001537+0000) 2024-03-18T01:52:11.571 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:52:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:52:11.282+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:51:28.001537+0000 (oldest deadline 2024-03-18T01:51:48.001537+0000) 2024-03-18T01:52:11.571 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:52:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:52:11.282+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:52:12.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:52:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:52:11.764+0000 7fa2c284a700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T01:52:12.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:52:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:52:11.764+0000 7fa2c284a700 -1 osd.1 46 *** Got signal Terminated *** 2024-03-18T01:52:12.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:52:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:52:11.764+0000 7fa2c284a700 -1 osd.1 46 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-18T01:52:12.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:52:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:52:11.809+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:51:15.812746+0000 (oldest deadline 2024-03-18T01:51:35.812746+0000) 2024-03-18T01:52:12.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:52:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:52:11.809+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:51:15.812746+0000 (oldest deadline 2024-03-18T01:51:35.812746+0000) 2024-03-18T01:52:12.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:52:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:52:11.809+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:51:15.812746+0000 (oldest deadline 2024-03-18T01:51:35.812746+0000) 2024-03-18T01:52:12.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:52:11 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:52:11.809+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:52:12.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:52:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:52:12.256+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:51:28.001537+0000 (oldest deadline 2024-03-18T01:51:48.001537+0000) 2024-03-18T01:52:12.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:52:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:52:12.256+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:51:28.001537+0000 (oldest deadline 2024-03-18T01:51:48.001537+0000) 2024-03-18T01:52:12.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:52:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:52:12.256+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:51:28.001537+0000 (oldest deadline 2024-03-18T01:51:48.001537+0000) 2024-03-18T01:52:12.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:52:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:52:12.257+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:52:13.302 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:52:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:52:12.830+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:51:15.812746+0000 (oldest deadline 2024-03-18T01:51:35.812746+0000) 2024-03-18T01:52:13.302 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:52:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:52:12.830+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:51:15.812746+0000 (oldest deadline 2024-03-18T01:51:35.812746+0000) 2024-03-18T01:52:13.302 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:52:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:52:12.830+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:51:15.812746+0000 (oldest deadline 2024-03-18T01:51:35.812746+0000) 2024-03-18T01:52:13.302 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:52:12 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:52:12.831+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:52:13.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:52:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:52:13.299+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:51:28.001537+0000 (oldest deadline 2024-03-18T01:51:48.001537+0000) 2024-03-18T01:52:13.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:52:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:52:13.299+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:51:28.001537+0000 (oldest deadline 2024-03-18T01:51:48.001537+0000) 2024-03-18T01:52:13.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:52:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:52:13.299+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:51:28.001537+0000 (oldest deadline 2024-03-18T01:51:48.001537+0000) 2024-03-18T01:52:13.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:52:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:52:13.299+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:52:14.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:52:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:52:13.788+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:51:15.812746+0000 (oldest deadline 2024-03-18T01:51:35.812746+0000) 2024-03-18T01:52:14.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:52:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:52:13.788+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:51:15.812746+0000 (oldest deadline 2024-03-18T01:51:35.812746+0000) 2024-03-18T01:52:14.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:52:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:52:13.788+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:51:15.812746+0000 (oldest deadline 2024-03-18T01:51:35.812746+0000) 2024-03-18T01:52:14.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:52:13 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:52:13.789+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:52:14.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:52:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:52:14.313+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:51:28.001537+0000 (oldest deadline 2024-03-18T01:51:48.001537+0000) 2024-03-18T01:52:14.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:52:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:52:14.313+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:51:28.001537+0000 (oldest deadline 2024-03-18T01:51:48.001537+0000) 2024-03-18T01:52:14.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:52:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:52:14.313+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:51:28.001537+0000 (oldest deadline 2024-03-18T01:51:48.001537+0000) 2024-03-18T01:52:14.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:52:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:52:14.313+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:52:15.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:52:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:52:14.775+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:51:15.812746+0000 (oldest deadline 2024-03-18T01:51:35.812746+0000) 2024-03-18T01:52:15.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:52:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:52:14.775+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:51:15.812746+0000 (oldest deadline 2024-03-18T01:51:35.812746+0000) 2024-03-18T01:52:15.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:52:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:52:14.775+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:51:15.812746+0000 (oldest deadline 2024-03-18T01:51:35.812746+0000) 2024-03-18T01:52:15.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:52:14 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:52:14.775+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:52:15.762 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:52:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:52:15.363+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:51:28.001537+0000 (oldest deadline 2024-03-18T01:51:48.001537+0000) 2024-03-18T01:52:15.763 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:52:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:52:15.363+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:51:28.001537+0000 (oldest deadline 2024-03-18T01:51:48.001537+0000) 2024-03-18T01:52:15.763 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:52:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:52:15.363+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:51:28.001537+0000 (oldest deadline 2024-03-18T01:51:48.001537+0000) 2024-03-18T01:52:15.763 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:52:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:52:15.363+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:52:16.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:52:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:52:15.761+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:51:15.812746+0000 (oldest deadline 2024-03-18T01:51:35.812746+0000) 2024-03-18T01:52:16.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:52:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:52:15.761+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:51:15.812746+0000 (oldest deadline 2024-03-18T01:51:35.812746+0000) 2024-03-18T01:52:16.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:52:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:52:15.761+0000 7fa2bee48700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:51:15.812746+0000 (oldest deadline 2024-03-18T01:51:35.812746+0000) 2024-03-18T01:52:16.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:52:15 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1[55037]: 2024-03-18T01:52:15.762+0000 7fa2bee48700 -1 osd.1 46 get_health_metrics reporting 227 slow ops, oldest is osd_op(mds.14480.15:258 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=36374b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-18T01:52:16.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:52:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:52:16.345+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:51:28.001537+0000 (oldest deadline 2024-03-18T01:51:48.001537+0000) 2024-03-18T01:52:16.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:52:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:52:16.345+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:51:28.001537+0000 (oldest deadline 2024-03-18T01:51:48.001537+0000) 2024-03-18T01:52:16.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:52:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:52:16.345+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:51:28.001537+0000 (oldest deadline 2024-03-18T01:51:48.001537+0000) 2024-03-18T01:52:16.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:52:16 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:52:16.345+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:52:17.538 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:52:17 smithi029 podman[85369]: 2024-03-18 01:52:17.278791419 +0000 UTC m=+5.637112725 container died f8915708098af4748229223efdbd2b845dc597734ef1cc963aa8a73c5e624655 (image=quay.ceph.io/ceph-ci/ceph@sha256:a75c953aa7dcb8597f235a84417a8be589209dba39db0305cebbb10d212afaf9, name=ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=quincy-6716c9f, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux ) 2024-03-18T01:52:17.539 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:52:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:52:17.355+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:51:28.001537+0000 (oldest deadline 2024-03-18T01:51:48.001537+0000) 2024-03-18T01:52:17.539 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:52:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:52:17.355+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:51:28.001537+0000 (oldest deadline 2024-03-18T01:51:48.001537+0000) 2024-03-18T01:52:17.539 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:52:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:52:17.355+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:51:28.001537+0000 (oldest deadline 2024-03-18T01:51:48.001537+0000) 2024-03-18T01:52:17.539 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:52:17 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:52:17.356+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:52:17.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:52:17 smithi029 podman[85369]: 2024-03-18 01:52:17.637005218 +0000 UTC m=+5.995326524 container cleanup f8915708098af4748229223efdbd2b845dc597734ef1cc963aa8a73c5e624655 (image=quay.ceph.io/ceph-ci/ceph@sha256:a75c953aa7dcb8597f235a84417a8be589209dba39db0305cebbb10d212afaf9, name=ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1, ceph=True, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=quincy-6716c9f, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2) 2024-03-18T01:52:17.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:52:17 smithi029 bash[85369]: ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1 2024-03-18T01:52:18.256 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:52:17 smithi029 podman[85387]: 2024-03-18 01:52:17.931108203 +0000 UTC m=+0.652031425 container remove f8915708098af4748229223efdbd2b845dc597734ef1cc963aa8a73c5e624655 (image=quay.ceph.io/ceph-ci/ceph@sha256:a75c953aa7dcb8597f235a84417a8be589209dba39db0305cebbb10d212afaf9, name=ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=quincy-6716c9f, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD) 2024-03-18T01:52:18.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:52:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:52:18.319+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:51:28.001537+0000 (oldest deadline 2024-03-18T01:51:48.001537+0000) 2024-03-18T01:52:18.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:52:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:52:18.319+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:51:28.001537+0000 (oldest deadline 2024-03-18T01:51:48.001537+0000) 2024-03-18T01:52:18.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:52:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:52:18.319+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:51:28.001537+0000 (oldest deadline 2024-03-18T01:51:48.001537+0000) 2024-03-18T01:52:18.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:52:18 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:52:18.319+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:52:18.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:52:18 smithi029 podman[85481]: 2024-03-18 01:52:18.158495718 +0000 UTC m=+0.016958812 image pull 19857db3290c58169e44a8aec8ef9646981a77ffd5b146ecef1b6faf59d6c3ab quay.ceph.io/ceph-ci/ceph@sha256:a75c953aa7dcb8597f235a84417a8be589209dba39db0305cebbb10d212afaf9 2024-03-18T01:52:18.571 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:52:18 smithi029 podman[85481]: 2024-03-18 01:52:18.257732264 +0000 UTC m=+0.116195391 container create ef06069143e2764a078eac84069229363acb902913ec441ccfe79ce01e7a3594 (image=quay.ceph.io/ceph-ci/ceph@sha256:a75c953aa7dcb8597f235a84417a8be589209dba39db0305cebbb10d212afaf9, name=ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1-deactivate, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=quincy-6716c9f, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, ceph=True) 2024-03-18T01:52:18.571 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:52:18 smithi029 podman[85481]: 2024-03-18 01:52:18.488988503 +0000 UTC m=+0.347451594 container init ef06069143e2764a078eac84069229363acb902913ec441ccfe79ce01e7a3594 (image=quay.ceph.io/ceph-ci/ceph@sha256:a75c953aa7dcb8597f235a84417a8be589209dba39db0305cebbb10d212afaf9, name=ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1-deactivate, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=quincy-6716c9f, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux ) 2024-03-18T01:52:18.571 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:52:18 smithi029 podman[85481]: 2024-03-18 01:52:18.492149226 +0000 UTC m=+0.350612312 container start ef06069143e2764a078eac84069229363acb902913ec441ccfe79ce01e7a3594 (image=quay.ceph.io/ceph-ci/ceph@sha256:a75c953aa7dcb8597f235a84417a8be589209dba39db0305cebbb10d212afaf9, name=ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=quincy-6716c9f, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True) 2024-03-18T01:52:19.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:52:18 smithi029 podman[85481]: 2024-03-18 01:52:18.605975 +0000 UTC m=+0.464438112 container attach ef06069143e2764a078eac84069229363acb902913ec441ccfe79ce01e7a3594 (image=quay.ceph.io/ceph-ci/ceph@sha256:a75c953aa7dcb8597f235a84417a8be589209dba39db0305cebbb10d212afaf9, name=ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1-deactivate, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, RELEASE=quincy-6716c9f, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , ceph=True) 2024-03-18T01:52:19.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:52:18 smithi029 podman[85481]: 2024-03-18 01:52:18.776135837 +0000 UTC m=+0.634598915 container died ef06069143e2764a078eac84069229363acb902913ec441ccfe79ce01e7a3594 (image=quay.ceph.io/ceph-ci/ceph@sha256:a75c953aa7dcb8597f235a84417a8be589209dba39db0305cebbb10d212afaf9, name=ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1-deactivate, GIT_CLEAN=True, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, RELEASE=quincy-6716c9f, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2) 2024-03-18T01:52:19.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:52:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:52:19.283+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:51:28.001537+0000 (oldest deadline 2024-03-18T01:51:48.001537+0000) 2024-03-18T01:52:19.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:52:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:52:19.283+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:51:28.001537+0000 (oldest deadline 2024-03-18T01:51:48.001537+0000) 2024-03-18T01:52:19.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:52:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:52:19.283+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:51:28.001537+0000 (oldest deadline 2024-03-18T01:51:48.001537+0000) 2024-03-18T01:52:19.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:52:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:52:19.283+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:52:19.810 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:52:19 smithi029 systemd[1]: Stopping Ceph osd.2 for 37f51182-e4c4-11ee-95c9-87774f69a715... 2024-03-18T01:52:19.810 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:52:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:52:19.809+0000 7f025e6da700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T01:52:19.811 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:52:19 smithi029 podman[85507]: 2024-03-18 01:52:19.339030255 +0000 UTC m=+0.556594061 container remove ef06069143e2764a078eac84069229363acb902913ec441ccfe79ce01e7a3594 (image=quay.ceph.io/ceph-ci/ceph@sha256:a75c953aa7dcb8597f235a84417a8be589209dba39db0305cebbb10d212afaf9, name=ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-1-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=quincy-6716c9f, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-18T01:52:19.811 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:52:19 smithi029 systemd[1]: ceph-37f51182-e4c4-11ee-95c9-87774f69a715@osd.1.service: Deactivated successfully. 2024-03-18T01:52:19.811 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:52:19 smithi029 systemd[1]: Stopped Ceph osd.1 for 37f51182-e4c4-11ee-95c9-87774f69a715. 2024-03-18T01:52:19.811 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 18 01:52:19 smithi029 systemd[1]: ceph-37f51182-e4c4-11ee-95c9-87774f69a715@osd.1.service: Consumed 17.270s CPU time. 2024-03-18T01:52:20.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:52:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:52:19.809+0000 7f025e6da700 -1 osd.2 46 *** Got signal Terminated *** 2024-03-18T01:52:20.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:52:19 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:52:19.809+0000 7f025e6da700 -1 osd.2 46 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-18T01:52:20.569 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:52:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:52:20.318+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:51:28.001537+0000 (oldest deadline 2024-03-18T01:51:48.001537+0000) 2024-03-18T01:52:20.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:52:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:52:20.318+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:51:28.001537+0000 (oldest deadline 2024-03-18T01:51:48.001537+0000) 2024-03-18T01:52:20.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:52:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:52:20.318+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:51:28.001537+0000 (oldest deadline 2024-03-18T01:51:48.001537+0000) 2024-03-18T01:52:20.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:52:20 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:52:20.318+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:52:21.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:52:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:52:21.344+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:51:28.001537+0000 (oldest deadline 2024-03-18T01:51:48.001537+0000) 2024-03-18T01:52:21.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:52:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:52:21.344+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:51:28.001537+0000 (oldest deadline 2024-03-18T01:51:48.001537+0000) 2024-03-18T01:52:21.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:52:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:52:21.344+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:51:28.001537+0000 (oldest deadline 2024-03-18T01:51:48.001537+0000) 2024-03-18T01:52:21.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:52:21 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:52:21.344+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:52:22.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:52:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:52:22.379+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:51:28.001537+0000 (oldest deadline 2024-03-18T01:51:48.001537+0000) 2024-03-18T01:52:22.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:52:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:52:22.379+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:51:28.001537+0000 (oldest deadline 2024-03-18T01:51:48.001537+0000) 2024-03-18T01:52:22.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:52:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:52:22.379+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:51:28.001537+0000 (oldest deadline 2024-03-18T01:51:48.001537+0000) 2024-03-18T01:52:22.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:52:22 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:52:22.379+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:52:23.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:52:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:52:23.378+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:51:28.001537+0000 (oldest deadline 2024-03-18T01:51:48.001537+0000) 2024-03-18T01:52:23.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:52:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:52:23.378+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:51:28.001537+0000 (oldest deadline 2024-03-18T01:51:48.001537+0000) 2024-03-18T01:52:23.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:52:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:52:23.378+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:51:28.001537+0000 (oldest deadline 2024-03-18T01:51:48.001537+0000) 2024-03-18T01:52:23.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:52:23 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:52:23.378+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:52:24.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:52:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:52:24.426+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-03-18T01:51:28.001537+0000 (oldest deadline 2024-03-18T01:51:48.001537+0000) 2024-03-18T01:52:24.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:52:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:52:24.426+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-03-18T01:51:28.001537+0000 (oldest deadline 2024-03-18T01:51:48.001537+0000) 2024-03-18T01:52:24.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:52:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:52:24.426+0000 7f025c4db700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-03-18T01:51:28.001537+0000 (oldest deadline 2024-03-18T01:51:48.001537+0000) 2024-03-18T01:52:24.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:52:24 smithi029 ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2[62401]: 2024-03-18T01:52:24.426+0000 7f025c4db700 -1 osd.2 46 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.24329.0:2902 3.8 3:109078f3:::2000000067e.00000000:head [write 0~1306 in=1306b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-18T01:52:25.569 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:52:25 smithi029 podman[85587]: 2024-03-18 01:52:25.251504991 +0000 UTC m=+5.560525681 container died cbf4647c043d69b3271b810ee2c2451ce96286e351e6639b20b140561bb79e6a (image=quay.ceph.io/ceph-ci/ceph@sha256:a75c953aa7dcb8597f235a84417a8be589209dba39db0305cebbb10d212afaf9, name=ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=quincy-6716c9f, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2024-03-18T01:52:26.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:52:25 smithi029 podman[85587]: 2024-03-18 01:52:25.57008359 +0000 UTC m=+5.879104280 container cleanup cbf4647c043d69b3271b810ee2c2451ce96286e351e6639b20b140561bb79e6a (image=quay.ceph.io/ceph-ci/ceph@sha256:a75c953aa7dcb8597f235a84417a8be589209dba39db0305cebbb10d212afaf9, name=ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2, RELEASE=quincy-6716c9f, ceph=True, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-18T01:52:26.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:52:25 smithi029 bash[85587]: ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2 2024-03-18T01:52:26.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:52:25 smithi029 podman[85607]: 2024-03-18 01:52:25.77427783 +0000 UTC m=+0.522077277 container remove cbf4647c043d69b3271b810ee2c2451ce96286e351e6639b20b140561bb79e6a (image=quay.ceph.io/ceph-ci/ceph@sha256:a75c953aa7dcb8597f235a84417a8be589209dba39db0305cebbb10d212afaf9, name=ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=quincy-6716c9f, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-03-18T01:52:26.447 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:52:26 smithi029 podman[85700]: 2024-03-18 01:52:26.096143375 +0000 UTC m=+0.018097068 image pull 19857db3290c58169e44a8aec8ef9646981a77ffd5b146ecef1b6faf59d6c3ab quay.ceph.io/ceph-ci/ceph@sha256:a75c953aa7dcb8597f235a84417a8be589209dba39db0305cebbb10d212afaf9 2024-03-18T01:52:26.447 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:52:26 smithi029 podman[85700]: 2024-03-18 01:52:26.209284136 +0000 UTC m=+0.131237856 container create c4e3088777368af493c20453f3ab1ec370b389de816381482c1bf343fda1b078 (image=quay.ceph.io/ceph-ci/ceph@sha256:a75c953aa7dcb8597f235a84417a8be589209dba39db0305cebbb10d212afaf9, name=ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2-deactivate, org.label-schema.vendor=CentOS, RELEASE=quincy-6716c9f, GIT_CLEAN=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307) 2024-03-18T01:52:26.447 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:52:26 smithi029 podman[85700]: 2024-03-18 01:52:26.446851207 +0000 UTC m=+0.368804896 container init c4e3088777368af493c20453f3ab1ec370b389de816381482c1bf343fda1b078 (image=quay.ceph.io/ceph-ci/ceph@sha256:a75c953aa7dcb8597f235a84417a8be589209dba39db0305cebbb10d212afaf9, name=ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2-deactivate, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , RELEASE=quincy-6716c9f, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-03-18T01:52:26.732 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:52:26 smithi029 podman[85700]: 2024-03-18 01:52:26.45051748 +0000 UTC m=+0.372471157 container start c4e3088777368af493c20453f3ab1ec370b389de816381482c1bf343fda1b078 (image=quay.ceph.io/ceph-ci/ceph@sha256:a75c953aa7dcb8597f235a84417a8be589209dba39db0305cebbb10d212afaf9, name=ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2-deactivate, org.label-schema.build-date=20240307, ceph=True, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=quincy-6716c9f, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-18T01:52:26.732 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:52:26 smithi029 podman[85700]: 2024-03-18 01:52:26.546886654 +0000 UTC m=+0.468840345 container attach c4e3088777368af493c20453f3ab1ec370b389de816381482c1bf343fda1b078 (image=quay.ceph.io/ceph-ci/ceph@sha256:a75c953aa7dcb8597f235a84417a8be589209dba39db0305cebbb10d212afaf9, name=ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2-deactivate, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , RELEASE=quincy-6716c9f, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-18T01:52:27.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:52:26 smithi029 podman[85700]: 2024-03-18 01:52:26.732360984 +0000 UTC m=+0.654314670 container died c4e3088777368af493c20453f3ab1ec370b389de816381482c1bf343fda1b078 (image=quay.ceph.io/ceph-ci/ceph@sha256:a75c953aa7dcb8597f235a84417a8be589209dba39db0305cebbb10d212afaf9, name=ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, RELEASE=quincy-6716c9f) 2024-03-18T01:52:27.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:52:27 smithi029 podman[85725]: 2024-03-18 01:52:27.35464533 +0000 UTC m=+0.614181481 container remove c4e3088777368af493c20453f3ab1ec370b389de816381482c1bf343fda1b078 (image=quay.ceph.io/ceph-ci/ceph@sha256:a75c953aa7dcb8597f235a84417a8be589209dba39db0305cebbb10d212afaf9, name=ceph-37f51182-e4c4-11ee-95c9-87774f69a715-osd-2-deactivate, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=quincy-6716c9f, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux ) 2024-03-18T01:52:27.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:52:27 smithi029 systemd[1]: ceph-37f51182-e4c4-11ee-95c9-87774f69a715@osd.2.service: Deactivated successfully. 2024-03-18T01:52:27.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:52:27 smithi029 systemd[1]: Stopped Ceph osd.2 for 37f51182-e4c4-11ee-95c9-87774f69a715. 2024-03-18T01:52:27.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 18 01:52:27 smithi029 systemd[1]: ceph-37f51182-e4c4-11ee-95c9-87774f69a715@osd.2.service: Consumed 15.239s CPU time. 2024-03-18T01:52:30.499 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2024-03-18T01:52:30.500 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-18T01:52:33.566 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.90 2024-03-18T01:52:42.568 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2024-03-18T01:52:42.569 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-18T01:52:45.634 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.90 2024-03-18T01:52:57.635 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2024-03-18T01:52:57.636 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-18T01:53:00.706 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.90 2024-03-18T01:53:03.707 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2024-03-18T01:53:03.707 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-18T01:53:03.778 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.90 2024-03-18T01:53:03.778 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-03-18T01:53:03.779 INFO:tasks.cephadm:Teardown complete 2024-03-18T01:53:03.779 ERROR:teuthology.run_tasks:Manager failed: cephadm Traceback (most recent call last): File "/home/teuthworker/src/github.com_batrick_ceph_c382027d5c66ef4e0aa1131ea3e8080c658d3184/qa/tasks/cephadm.py", line 1613, in distribute_config_and_admin_keyring yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_batrick_ceph_c382027d5c66ef4e0aa1131ea3e8080c658d3184/qa/tasks/cephadm.py", line 1867, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_batrick_ceph_c382027d5c66ef4e0aa1131ea3e8080c658d3184/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_batrick_ceph_c382027d5c66ef4e0aa1131ea3e8080c658d3184/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_batrick_ceph_c382027d5c66ef4e0aa1131ea3e8080c658d3184/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_batrick_ceph_c382027d5c66ef4e0aa1131ea3e8080c658d3184/qa/tasks/cephadm.py", line 1870, in task log.info('Teardown begin') File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_batrick_ceph_c382027d5c66ef4e0aa1131ea3e8080c658d3184/qa/tasks/cephadm.py", line 1746, in initialize_config yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_batrick_ceph_c382027d5c66ef4e0aa1131ea3e8080c658d3184/qa/tasks/cephadm.py", line 200, in download_cephadm _rm_cluster(ctx, cluster_name) File "/home/teuthworker/src/github.com_batrick_ceph_c382027d5c66ef4e0aa1131ea3e8080c658d3184/qa/tasks/cephadm.py", line 368, in _rm_cluster ctx.cluster.run(args=[ File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi090 2024-03-18T01:53:03.801 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-03-18T01:53:03.812 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/install/__init__.py", line 626, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_batrick_ceph_c382027d5c66ef4e0aa1131ea3e8080c658d3184/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_batrick_ceph_c382027d5c66ef4e0aa1131ea3e8080c658d3184/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_batrick_ceph_c382027d5c66ef4e0aa1131ea3e8080c658d3184/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host 2024-03-18T01:53:03.828 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-03-18T01:53:03.828 DEBUG:teuthology.orchestra.run.smithi029:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-18T01:53:03.841 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2024-03-18T01:53:03.841 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-18T01:53:06.910 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.90 2024-03-18T01:53:15.912 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2024-03-18T01:53:15.913 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-18T01:53:18.974 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.90 2024-03-18T01:53:30.975 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2024-03-18T01:53:30.976 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-18T01:53:34.046 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.90 2024-03-18T01:53:37.047 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2024-03-18T01:53:37.048 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-18T01:53:37.118 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.90 2024-03-18T01:53:37.118 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-03-18T01:53:37.165 ERROR:teuthology.run_tasks:Manager failed: install Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/install/util.py", line 148, in ship_utilities yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/install/__init__.py", line 626, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_batrick_ceph_c382027d5c66ef4e0aa1131ea3e8080c658d3184/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_batrick_ceph_c382027d5c66ef4e0aa1131ea3e8080c658d3184/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_batrick_ceph_c382027d5c66ef4e0aa1131ea3e8080c658d3184/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/install/__init__.py", line 626, in task yield File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/install/__init__.py", line 220, in install yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/install/util.py", line 150, in ship_utilities _remove_utilities(ctx, filenames) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/install/util.py", line 121, in _remove_utilities ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi090 2024-03-18T01:53:37.185 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-18T01:53:37.200 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-18T01:53:37.200 DEBUG:teuthology.orchestra.run.smithi029:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-18T01:53:37.208 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2024-03-18T01:53:37.209 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-18T01:53:37.224 INFO:teuthology.orchestra.run.smithi029.stderr:bash: line 1: ntpq: command not found 2024-03-18T01:53:37.228 INFO:teuthology.orchestra.run.smithi029.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-18T01:53:37.229 INFO:teuthology.orchestra.run.smithi029.stdout:=============================================================================== 2024-03-18T01:53:37.229 INFO:teuthology.orchestra.run.smithi029.stdout:^+ hv01.front.sepia.ceph.com 2 7 377 81 +3788us[+3698us] +/- 127ms 2024-03-18T01:53:37.229 INFO:teuthology.orchestra.run.smithi029.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 20 -2040us[-2145us] +/- 82ms 2024-03-18T01:53:37.229 INFO:teuthology.orchestra.run.smithi029.stdout:^* hv03.front.sepia.ceph.com 3 6 377 17 -324us[ -429us] +/- 69ms 2024-03-18T01:53:37.229 INFO:teuthology.orchestra.run.smithi029.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-03-18T01:53:40.286 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.90 2024-03-18T01:53:49.287 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2024-03-18T01:53:49.288 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-18T01:53:52.350 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.90 2024-03-18T01:54:04.351 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2024-03-18T01:54:04.352 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-18T01:54:07.422 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.90 2024-03-18T01:54:10.423 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2024-03-18T01:54:10.424 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-18T01:54:10.494 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.90 2024-03-18T01:54:10.494 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-03-18T01:54:10.495 ERROR:teuthology.run_tasks:Manager failed: clock Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/clock.py", line 66, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_batrick_ceph_c382027d5c66ef4e0aa1131ea3e8080c658d3184/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_batrick_ceph_c382027d5c66ef4e0aa1131ea3e8080c658d3184/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_batrick_ceph_c382027d5c66ef4e0aa1131ea3e8080c658d3184/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/clock.py", line 72, in task cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi090 2024-03-18T01:54:10.509 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-18T01:54:10.521 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-18T01:54:10.521 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-18T01:54:10.533 DEBUG:teuthology.orchestra.run.smithi029:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-18T01:54:10.621 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2024-03-18T01:54:10.622 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-18T01:54:13.694 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.90 2024-03-18T01:54:22.695 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2024-03-18T01:54:22.696 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-18T01:54:25.758 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.90 2024-03-18T01:54:37.759 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2024-03-18T01:54:37.760 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-18T01:54:40.834 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.90 2024-03-18T01:54:43.834 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2024-03-18T01:54:43.835 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-18T01:54:43.902 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.90 2024-03-18T01:54:43.903 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-03-18T01:54:43.903 ERROR:teuthology.run_tasks:Manager failed: selinux Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/__init__.py", line 136, in __exit__ self.teardown() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/selinux.py", line 165, in teardown self.archive_log() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/selinux.py", line 195, in archive_log self.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi090 2024-03-18T01:54:43.911 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-18T01:54:43.922 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-18T01:54:43.932 INFO:teuthology.task.internal:Duration was 3135.821147 seconds 2024-03-18T01:54:43.933 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-18T01:54:43.943 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-18T01:54:43.944 DEBUG:teuthology.orchestra.run.smithi029:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-18T01:54:43.949 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2024-03-18T01:54:43.950 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-18T01:54:43.997 INFO:teuthology.orchestra.run.smithi029.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-18T01:54:47.006 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.90 2024-03-18T01:54:56.007 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2024-03-18T01:54:56.008 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-18T01:54:59.070 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.90 2024-03-18T01:55:11.071 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2024-03-18T01:55:11.072 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-18T01:55:14.142 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.90 2024-03-18T01:55:17.143 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2024-03-18T01:55:17.143 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-18T01:55:17.218 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.90 2024-03-18T01:55:17.218 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-03-18T01:55:17.219 ERROR:teuthology.run_tasks:Manager failed: internal.syslog Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/internal/syslog.py", line 76, in syslog yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_batrick_ceph_c382027d5c66ef4e0aa1131ea3e8080c658d3184/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_batrick_ceph_c382027d5c66ef4e0aa1131ea3e8080c658d3184/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_batrick_ceph_c382027d5c66ef4e0aa1131ea3e8080c658d3184/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/internal/syslog.py", line 85, in syslog cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi090 2024-03-18T01:55:17.234 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-18T01:55:17.248 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-18T01:55:17.248 DEBUG:teuthology.orchestra.run.smithi029:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-18T01:55:17.284 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2024-03-18T01:55:17.285 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-18T01:55:20.350 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.90 2024-03-18T01:55:29.351 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2024-03-18T01:55:29.352 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-18T01:55:32.414 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.90 2024-03-18T01:55:44.415 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2024-03-18T01:55:44.416 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-18T01:55:47.486 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.90 2024-03-18T01:55:50.487 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2024-03-18T01:55:50.488 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-18T01:55:50.558 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.90 2024-03-18T01:55:50.558 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-03-18T01:55:50.559 ERROR:teuthology.run_tasks:Manager failed: internal.sudo Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/internal/__init__.py", line 452, in sudo yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_batrick_ceph_c382027d5c66ef4e0aa1131ea3e8080c658d3184/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_batrick_ceph_c382027d5c66ef4e0aa1131ea3e8080c658d3184/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_batrick_ceph_c382027d5c66ef4e0aa1131ea3e8080c658d3184/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/internal/__init__.py", line 455, in sudo ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi090 2024-03-18T01:55:50.576 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-18T01:55:50.588 DEBUG:teuthology.orchestra.run.smithi029:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-18T01:55:50.595 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2024-03-18T01:55:50.596 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-18T01:55:50.623 INFO:teuthology.orchestra.run.smithi029.stdout:kernel.core_pattern = core 2024-03-18T01:55:53.662 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.90 2024-03-18T01:56:02.663 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2024-03-18T01:56:02.664 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-18T01:56:05.726 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.90 2024-03-18T01:56:17.727 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2024-03-18T01:56:17.728 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-18T01:56:20.798 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.90 2024-03-18T01:56:23.799 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2024-03-18T01:56:23.801 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-18T01:56:23.874 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.90 2024-03-18T01:56:23.874 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-03-18T01:56:23.875 ERROR:teuthology.run_tasks:Manager failed: internal.coredump Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/internal/__init__.py", line 488, in coredump yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_batrick_ceph_c382027d5c66ef4e0aa1131ea3e8080c658d3184/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_batrick_ceph_c382027d5c66ef4e0aa1131ea3e8080c658d3184/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_batrick_ceph_c382027d5c66ef4e0aa1131ea3e8080c658d3184/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/internal/__init__.py", line 492, in coredump cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi090 2024-03-18T01:56:23.892 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-18T01:56:23.904 INFO:teuthology.task.internal:Transferring archived files... 2024-03-18T01:56:23.905 DEBUG:teuthology.misc:Transferring archived files from smithi029:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/pdonnell-2024-03-17_23:17:27-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7608561/remote/smithi029 2024-03-18T01:56:23.906 DEBUG:teuthology.orchestra.run.smithi029:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-18T01:56:23.948 DEBUG:teuthology.misc:Transferring archived files from smithi090:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/pdonnell-2024-03-17_23:17:27-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7608561/remote/smithi090 2024-03-18T01:56:23.948 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2024-03-18T01:56:23.949 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-18T01:56:27.006 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.90 2024-03-18T01:56:36.007 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2024-03-18T01:56:36.008 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-18T01:56:39.070 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.90 2024-03-18T01:56:51.071 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2024-03-18T01:56:51.072 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-18T01:56:54.142 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.90 2024-03-18T01:56:57.144 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2024-03-18T01:56:57.145 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-18T01:56:57.214 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.90 2024-03-18T01:56:57.214 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-03-18T01:56:57.215 ERROR:teuthology.run_tasks:Manager failed: internal.archive Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/internal/__init__.py", line 392, in archive yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_batrick_ceph_c382027d5c66ef4e0aa1131ea3e8080c658d3184/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_batrick_ceph_c382027d5c66ef4e0aa1131ea3e8080c658d3184/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_batrick_ceph_c382027d5c66ef4e0aa1131ea3e8080c658d3184/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/internal/__init__.py", line 418, in archive misc.pull_directory(rem, archive_dir, path, maybe_compress) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 748, in pull_directory r = remote.get_tar_stream(remotedir, sudo=True, compress=False) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 678, in get_tar_stream return self.run(args=args, wait=False, stdout=run.PIPE) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi090 2024-03-18T01:56:57.232 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-18T01:56:57.245 INFO:teuthology.task.internal:Not uploading archives. 2024-03-18T01:56:57.245 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-18T01:56:57.256 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-18T01:56:57.257 DEBUG:teuthology.orchestra.run.smithi029:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-18T01:56:57.263 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2024-03-18T01:56:57.264 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-18T01:56:57.279 INFO:teuthology.orchestra.run.smithi029.stdout: 265300 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 18 01:53 /home/ubuntu/cephtest 2024-03-18T01:56:57.279 INFO:teuthology.orchestra.run.smithi029.stdout: 265301 4 drwxr-xr-x 4 ubuntu ubuntu 4096 Mar 18 01:55 /home/ubuntu/cephtest/archive 2024-03-18T01:56:57.279 INFO:teuthology.orchestra.run.smithi029.stdout: 265305 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 18 01:02 /home/ubuntu/cephtest/archive/syslog 2024-03-18T01:56:57.279 INFO:teuthology.orchestra.run.smithi029.stdout: 265307 0 -rw-rw-rw- 1 ubuntu ubuntu 0 Mar 18 01:02 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-18T01:56:57.279 INFO:teuthology.orchestra.run.smithi029.stdout: 265306 0 -rw-rw-rw- 1 ubuntu ubuntu 0 Mar 18 01:02 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-18T01:56:57.279 INFO:teuthology.orchestra.run.smithi029.stdout: 402361 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 18 01:54 /home/ubuntu/cephtest/archive/audit 2024-03-18T01:56:57.279 INFO:teuthology.orchestra.run.smithi029.stdout: 426279 180 -rw------- 1 ubuntu ubuntu 180855 Mar 18 01:54 /home/ubuntu/cephtest/archive/audit/audit.log.gz 2024-03-18T01:56:57.280 INFO:teuthology.orchestra.run.smithi029.stdout: 265391 368 -rwxr-xr-x 1 ubuntu ubuntu 375153 Mar 18 01:09 /home/ubuntu/cephtest/cephadm 2024-03-18T01:56:57.280 INFO:teuthology.orchestra.run.smithi029.stderr:rmdir: failed to remove '/home/ubuntu/cephtest': Directory not empty 2024-03-18T01:57:00.318 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.90 2024-03-18T01:57:09.320 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2024-03-18T01:57:09.321 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-18T01:57:12.386 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.90 2024-03-18T01:57:24.388 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2024-03-18T01:57:24.388 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-18T01:57:27.458 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.90 2024-03-18T01:57:30.459 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2024-03-18T01:57:30.460 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-18T01:57:30.530 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.90 2024-03-18T01:57:30.530 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-03-18T01:57:30.531 ERROR:teuthology.run_tasks:Manager failed: internal.base Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/internal/__init__.py", line 48, in base yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_batrick_ceph_c382027d5c66ef4e0aa1131ea3e8080c658d3184/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_batrick_ceph_c382027d5c66ef4e0aa1131ea3e8080c658d3184/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_batrick_ceph_c382027d5c66ef4e0aa1131ea3e8080c658d3184/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/internal/__init__.py", line 54, in base ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi090 2024-03-18T01:57:30.548 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-18T01:57:30.560 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-18T01:57:30.573 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: OSError: [Errno 113] No route to host 2024-03-18T01:57:30.573 INFO:teuthology.nuke:Checking targets against current locks 2024-03-18T01:57:30.607 DEBUG:teuthology.nuke:shortname: smithi029 2024-03-18T01:57:30.607 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-18T01:57:30.633 DEBUG:teuthology.nuke:shortname: smithi090 2024-03-18T01:57:30.634 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-18T01:57:30.637 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi029.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-03-17_23:17:27-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7608561', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-18 00:53:09.031076', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGW25UlkH/bXu4KCYGfVm5U2VBzpkQLaPDsijUdeDame6Fwc6a3NjBriD+CKqSrhlp3TjuC2OL0SiPDlE1QnNwk='} 2024-03-18T01:57:30.660 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi090.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-03-17_23:17:27-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7608561', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-18 00:53:09.029423', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHzv2Sz8HwWmBAuxDXa5G3WXiAPTu2EWfpwaaLQLCCw2tsnu16FDoyG4iehdAol9ADDbmrI0DVahcmM7kKti0Z8='} 2024-03-18T01:57:30.690 INFO:teuthology.orchestra.console.smithi029:Power off 2024-03-18T01:57:30.690 DEBUG:teuthology.orchestra.console.smithi029:pexpect command: ipmitool -H smithi029.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-18T01:57:30.714 DEBUG:teuthology.orchestra.console.smithi029:power off output: Chassis Power Control: Down/Off 2024-03-18T01:57:30.715 DEBUG:teuthology.orchestra.console.smithi029:pexpect command: ipmitool -H smithi029.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-18T01:57:30.732 INFO:teuthology.orchestra.console.smithi090:Power off 2024-03-18T01:57:30.732 DEBUG:teuthology.orchestra.console.smithi090:pexpect command: ipmitool -H smithi090.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-18T01:57:30.743 DEBUG:teuthology.orchestra.console.smithi029:check power output: Chassis Power is on 2024-03-18T01:57:30.756 DEBUG:teuthology.orchestra.console.smithi090:power off output: Chassis Power Control: Down/Off 2024-03-18T01:57:30.756 DEBUG:teuthology.orchestra.console.smithi090:pexpect command: ipmitool -H smithi090.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-18T01:57:30.785 DEBUG:teuthology.orchestra.console.smithi090:check power output: Chassis Power is on 2024-03-18T01:57:34.744 DEBUG:teuthology.orchestra.console.smithi029:pexpect command: ipmitool -H smithi029.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-18T01:57:34.787 DEBUG:teuthology.orchestra.console.smithi090:pexpect command: ipmitool -H smithi090.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-18T01:57:34.854 DEBUG:teuthology.orchestra.console.smithi029:check power output: Chassis Power is on 2024-03-18T01:57:34.897 DEBUG:teuthology.orchestra.console.smithi090:check power output: Chassis Power is on 2024-03-18T01:57:38.860 DEBUG:teuthology.orchestra.console.smithi029:pexpect command: ipmitool -H smithi029.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-18T01:57:38.898 DEBUG:teuthology.orchestra.console.smithi090:pexpect command: ipmitool -H smithi090.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-18T01:57:38.972 DEBUG:teuthology.orchestra.console.smithi029:check power output: Chassis Power is off 2024-03-18T01:57:39.009 DEBUG:teuthology.orchestra.console.smithi090:check power output: Chassis Power is off 2024-03-18T01:57:39.073 INFO:teuthology.orchestra.console.smithi029:Power off completed 2024-03-18T01:57:39.110 INFO:teuthology.orchestra.console.smithi090:Power off completed 2024-03-18T01:57:39.216 INFO:teuthology.run:Summary data: description: fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} duration: 3135.8211467266083 failure_reason: 'Command failed on smithi029 with status 1: "sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 37f51182-e4c4-11ee-95c9-87774f69a715 -e sha1=3d78f5ba07bf18880172c07d4d7b4ac00c9d30af -- bash -c ''ceph orch ps''"' flavor: default owner: scheduled_pdonnell@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=d23aae5f1e6446eba31826c0f3a4cce6 status: fail success: false 2024-03-18T01:57:39.216 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-18T01:57:39.298 INFO:teuthology.run:FAIL